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Abstract 

The  purpose  of  this  research  was  to  identify  meaningful 
measures  (categories)  of  responsiveness  for  A950th  Test  Wing 
managers  and  System  Program  Office/Laboratory  managers  to 
use  to  evaluate  Test  Wing  effectiveness  and  responsiveness 
in  meeting  customer  test  program  information  needs. 

The  study  found  that  Test  Wing  managers  have  available 
to  them  a  prescribed  list  of  categories  of  information  (from 
Appendix  H,  Test  Director's  Guide,  A950  TESTWP  30-1)  through 
which  they  can  report  test  program  information  to  their 
customers.  Customers  such  as  the  Aeronautical  Systems 
Division  System  Program  Offices  and  the  Wright  Research  and 
Development  Center  Laboratories  differ  from  Test  Wing 
managers  in  their  perceptions  about  which  categories  are 
applicable  to  their  programs.  Test  Wing  managers  and  their 
customers  value  the  importance  of  the  categories  of 
information  similarly  for  most  of  the  categories.  Although 
many  perceptions  about  the  category  of  information 
applicability  and  importance  may  be  similar,  differences  do 
exist  and  for  a  variety  of  reasons. 

The  Test  Wing  was  rated  by  itself  and  its  customers  in 
how  flexible  they  were  perceived  to  be  in  negotiating  what 
and  how  information  for  each  category  was  to  be  reported  to 
customers.  Results  showed  that  Test  Wing  managers  and  their 
customers  have  similar  perceptions.  The  highest  ratings 
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were  received  for  the  categories  of  flight  testing, 
schedule,  planning,  technical  performance,  and 
instrumentation.  The  lowest  mean  ratings  occurred  for  the 
categories  of  cost,  manpower,  funding,  and  documentation. 

In  rating  Test  Wing  overall  reporting  performance  for 
each  category  of  information.  Test  Wing  managers  and  their 
customers  have  similar  perceptions.  The  Test  Wing  received 
highest  ratings  for  their  reporting  in  the  categories  of 
range  support,  flight  testing,  safety,  schedule,  and 
planning,  and  lowest  ratings  for  prefab,  modification 
installation,  and  manpower. 

Open-ended  comments  by  the  interview  respondents 
suggested  reform  in  the  methods  in  which  the  Test  Wing 
reports  cost  information. 


1 


THE  IDENTIFICATION  AND  EVALUATION  OF  FLIGHT  TEST  CATEGORIES 
OF  RESPONSIVENESS  IN  REPORTING  WEAPON  SYSTEM  TEST 
PROGRAM  INFORMATION  BY  THE 
4950TH  TEST  WING 

I.  Introduction  to  the  Study 

General  Issue 

The  flight  testing  of  newly  developed  or  modified 
weapon  systems  is  an  important  responsibility  of  tlie  Air 
Force.  The  challenge  arises  not  just  because  of  the 
comp lexity  of  today's  weapons  systems  but  also  because  of 
the  complexity  of  managing  and  reporting  test  program 
progress.  The  effective  management  of  a  test  program 
requires  accurate  and  timely  test  program  progress 
information  as  input  for  decisions  that  affect  the  test 
program.  The  reporting  of  test  program  progress  information 
is  done  by  an  Air  Force  agency  called  a  responsible  test 
organization  (RTO).  Tue  RTO  is  responsible  for  conducting 
the  flight  tests  and  reporting  test  data  and  test  program 
progress  information  to  its  cu3tomers--of ten  system  program 
offices  (SPO).  One  important  ingredient  of  an  effective 
test  program  is  the  program's  responsiveness  in  reporting 
test  information  to  its  customers.  Colonel  Michael  Baran, 
former  test  director  for  the  B-IB  bomber  program  and  former 
customer  of  an  RTO,  said  the  following  about  the  need  for 
effective  reporting  of  test  program  progress  information: 
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To  manage,  you  must  have  timely,  effective  feedback 
which  permits  you  to  assess  progress  and  diagnose 
problems.  .  .  .  The  Director  of  Test  must  establish  the 

responsibilities  and  data  flow  between  the  and  the 

RTO.  This  is  not  an  easy  task,  and  history  would 
indicate  it  is  one  that  has  seldom  been  done  well. 

( A  :  69  ) 

One  of  Air  Force  Systems  Command's  (AFSC)  RTOs ,  the 
A950th  Test  Wing,  has  expressed  a  need  for  development  of 
meaningful  measures  (or  categories)  of  responsiveness  for 
its  managers  to  use  to  determine  the  Test  Wing's 
responsiveness  in  meeting  customer  needs. 


Objectives  of  this  Research 

This  research  project  has  two  specific  objectives: 

1.  to  determine  the  similarities  and  differences  that 
may  exist  between  the  perceptions  of  the  A950th  Test  Wing 
and  its  customers  concerning  what  constitutes  effective 
reporting,  and 

2.  to  determine  the  similarities  and  differences  that 
may  exist  between  the  Test  Wing's  assessment  of  its 
performance  and  its  customers'  assessment  of  Test  Wing 
performance  in  reporting  information. 

The  research  involves  the  identification  of  meaningful 
measures  (or  catagories)  of  responsiveness  for  A950th  Test 
Wing  managers  and  SPO  managers  to  use  to  evaluate  Test  Wing 
effectiveness  and  responsiveness  in  meeting  customer  test 
program  progress  information  needs. 


Research  Questions 

The  following  investigative  questions  must  be  answered 
in  order  to  meet  the  two  research  objectives: 


-> 


1.  What  are  the  current  categories  of  information 
that  the  4950th  Test  Wing  repor ts--f rom  the  SPO  management 
perspective  and  from  the  Test  Wing  management  perspective? 

2.  What,  if  any  additional  categories  of  information 
do  the  Test  Wing  or  its  customers  believe  should  be 
reported? 

3.  What,  if  any,  categories  that  are  currently 
reported  do  either  the  Test  Wing  or  its  customers  feel 
should  not  be  reported? 

4.  What  are  the  relative  weights  assigned  to  the 
actual  and  desired  categories  by  SPO  management  and  Test 
Wing  management? 

5.  VJhat  are  the  most  common  methods  Test  Wing 
managers  use  in  reporting  test  information  for  each  category 
to  SPO  managers,  as  perceived  by  the  Test  Wing  and  by  its 
customers? 

6.  What,  if  any,  differences  in  reporting  practices 
would  SPO  managers  recommend? 

7.  What,  if  any,  differences  would  Test  Wing  managers 
prefer  to  make  In  reporting  information  to  their  customers? 


Research  Scope 

This  research  effort  will  consider  only  local  area 
4950th  Test  Wing  customers  from  Aeronautical  Systems 
Division  (ASD)  and  Wright  Research  and  Development  Center 
(WRDC).  Both  of  these  Test  Wing  customers  are  tenant 
organizations  based  at  Wr i ght-Pat ter son  Air  Force  Base 
(WPaFoI.  Until  recently,  WRDC  was  known  as  Air  Force  Wright- 
Aeronautical  Laboratories  (AFWAL).  Limitations  of  time  and 
travel  funds  prohibit  an  examination  of  Test  Wing  customers 
from  other  AF3C  product  divisions.  Although  the  research 
issues  of  interest  are  relevant  to  other  test  wings  and 
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product  divisions,  the  conclusions  of  this  research  effort 
will  be  specific  to  the  4950th  Test  Wing,  ASD,  and  WRDC . 
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Literature  Reviow 


II . 


Chapter  Overview 

This  chapter  presents  a  review  of  literature  on  the 
guidelines  and  policies  governing  the  reporting  of  test 
program  progress.  Background  about  information  gained  by 
testing  weapon  systems  is  discussed,  followed  by  a  brief 
explanation  of  the  Teat  and  Evaluation  (T&E)  process  that 
theoretically  occurs  between  System  Program  Offices  (SPOs) 
and  Responsible  Test  Organizations  (RTOs).  Regulations  and 
pamphlets  specifically  addressing  the  information  reporting 
policies  of  the  49S0th  Test  Wing  are  also  discussed. 

Background 

The  basic  purpose  of  testing  is  to  gain  information. 
Information  gained  from  testing  is  important  to  systems 
acquisition  program  managers.  These  program  managers  need 
test  information  in  order  to  make  competent  management 
decisions  related  to  the  timely  development,  production,  and 
fielding  of  systems  that  meet  the  user's  requirements  and 
are  operationally  effective  and  suitable  (8:2).  The 
different  types  of  information  yielded  by  testing  are  useful 
to  different  people  in  different  ways.  For  example,  weapon 
system  design  engineers  are  interested  in  the  quantitative 
flight  test  data  representing  the  weapon  system  performance. 
SPO  test  managers  are  interested  in  performance  data  as 
well,  but  in  their  function  as  the  managers  responsible  for 
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entire  test  programs,  they  are  also  concerned  with 
information  regarding  the  test  program  progress. 

Test  program  progress  is  achieved  by  the  test  pilots, 
engineers,  and  technicians  in  the  RTO.  Depending  on  the 
program,  a  private  contractor  may  also  be  involved  in 
performing  flight  tests.  The  progress  that  is  made  is 
reported  by  a  test  wing  manager  at  the  RTO  to  the  S PO  test 
manager.  As  alluded  to  earlier,  the  effectiveness  of  a  test 
program  is  dependent  on  the  program's  responsiveness  in 
reporting  test  information  to  its  customers. 

The  way  in  which  test  program  progress  is  reported  by 
the  RTO  depends  somewhat  on  the  agreement  that  is 
negotiated  between  the  SPO  and  the  RTO.  When  a  test  program 
is  initiated,  a  relationship  is  formally  established  between 
a  SPO  and  an  RTO  via  a  process  that  is  discussed  in  the  next 
section.  The  SPO  is  ultimately  responsible  for  the  overall 
test  program  and  has  overall  management  and  control 
responsibility.  In  a  sense,  the  RTO  serves  the  SPO  in  the 
capacity  of  a  contractor  by  performing  and  managing  the  day- 
to-day  flight  test  activities.  Since  the  SPO  test  manager 
is  responsible  for  the  success  of  the  test  program,  he  or 
she  must  implement  a  control  system  to  monitor  and 
(whenever  necessary  and  possible)  influence  the  activities 
of  the  RTO.  Adams  and  Martin  describe  the  need  for  such  a 
control  system  in  the  following  way: 

Control  systems  are  necessary  to  bring  this 

information  to  the  attention  of  the  project  [or  SPO 
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test]  manager  early  enough  to  avoid  a  crisis  by  means 
of  a  budget  or  some  other  project  change.  (1:51) 

If  timely  and  appropriate  information  about  the  test 

program  progress  can  be  reported  by  the  RTO  to  the  SPO 

within  the  control  system,  the  SPO  manager  can  compare  the 

reported  information  with  planned  performance  and  consider 

whether  change  to  the  baseline  plan  is  necessary  (1:52). 

Project  management  literature  identifies  three  critical 

categories  of  information  that  must  be  reported  to  managers: 

cost,  schedule,  and  performance  (1:53).  These  same  three 

categories  of  information  are  identified  in  the  Aeronautical 

Systems  Division  (ASD)  pamphlet,  ASDP  80-lA,  as  system 

technical  performance,  test  schedule,  and  test  support  cost 

(6:39). 

Although  reporting  information  in  the  categories  of 
cost,  schedule,  and  performance  fulfills  a  requirement  in 
satisfying  the  basic  control  system  needs  of  the  SPO  test 
manager,  these  categories  do  not  eracompass  all  of  the 
information  that  program  managers  need  for  effective 
management.  As  stated  in  the  objectives  section,  this 
research  effort  will,  among  other  things,  determine  what 
constitutes  effective  reporting  in  terms  of  categories  of 
responsiveness.  This  research  effort,  in  its  attempt  to 
identify  and  evaluate  the  categories  of  responsiveness,  will 
also  provide  an  assessment  of  how  satisfied  some  of  the  Test 
Wing's  customers  are  in  terms  of  the  categories  used. 
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Test  and  Evaluation  Process 


As  discussed  in  the  background  section,  the  basic 

pu 'pose  of  test  and  evaluation  in  any  process  is  to  gain 

information  that  will  assist  a  decision-maker  in  making 

competent  decisions.  In  the  Department  of  Defense 

acquisition  process,  test  and  evaluation  exists  in  two  main 

forms --deve lopmenta 1  test  and  evaluation  (commonly  called 

DT&.E)  and  operational  test  and  evaluation  (commonly  called 

OT&E)--to  provide  information  to  the  review  board 

responsible  for  making  program  milestone  decisions.  This 

research  effort  is  concerned  only  with  developmental  test 

programs.  DT&E  is  conducted  to  do  the  following; 

Assist  the  engineering  design  and  development 
process 

Verify  attainment  of  performance  specifications  and 
objectives 

Demonstrate  that  design  risks  have  been  minimized 
Estimate  the  system’s  military  utility  when 
introduced 

Evaluate  compatibility  and  interoperability  with 
existing  or  planned  equipment/systems 
Provide  assurance  that  the  equipment/system  is  ready 
for  testing  in  the  operational  environment  (5:13-3 
to  13-4) 

As  developmental  testing  of  a  weapon  system 
progresses,  information  is  obtained  and  reported  by  the 
testing  organization  (RTO)  to  its  customer  (SPO)  as  input  to 
the  process  of  managing  the  program.  The  Defense  Systems 
Management  College  (DSMC),  a  school  for  current  and 
potential  Department  of  Defense  (DoD)  acquisition  managers, 
identifies  the  need  for  program  managers  to  have  proper 
information  in  the  following: 
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The  Program  Manager  .  .  .  must  maintain  a  real-time 

network  that  provides  all  the  proper  information  with 
which  to  make  engineering  and  program  decisions. 

(5 ; 13-lA) 

At  a  macro  level,  test  program  reporting  is  received  at 
the  Secretary  of  Defense  or  Secretary  of  the  Air  Force 
level,  where  the  reporting  of  test  results  of  major  programs 
is  necessary  for  the  Defense  Acquisition  Board  (DAB)  or  the 
Air  Force  Systems  Acquisition  Review  Council  (AFSARC)  to 
make  major  milestone  decisions.  These  decisions  consist  of 
the  following  three  choices,  as  defined  in  the  readings  of 
the  Air  Force  Institute  of  Technology's  Test  and  Evaluation 
Management  course:  "(1)  approve  entry  into  the  next 
[acquisition]  phase,  (2)  request  further  Air  Force  studies, 
or  (3)  terminate  the  program"  (13:44). 

At  the  micro  level,  a  SPO  test  manager  will  take  test 
program  information  provided  by  the  RTO  and  make  decisions 
for  the  daily  management  of  the  test  program.  The  SPO  test 
manager  is  a  functional  manager  who  supports  the  program 
manager/d i rec tor  of  the  SPO.  As  a  supporting  manager,  the 
SPO  test  manager  provides  input  from  the  test  program  that 
influences  decisions  regarding  management  of  the  overall 
program . 

As  a  point  of  clarification,  this  research  effort 
focuses  on  the  micro  level  of  test  reporting  Instead  of  the 
macro  level.  Within  the  micro  level  of  test  reporting,  this 
research  focuses  on  an  RTO's  reporting  of  test  information 
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rather  than  an  RTO's  performance  as  a  weapon  system  tester. 
The  RTO  under  study  was  the  A950th  Test  Wing. 

The  way  in  which  the  reporting  process  is  established 
between  the  SPO  and  RTO  influences  how  information  is 
reported.  At  the  inception  of  a  program,  guidance  from 
Headquarters  USAF  is  provided  to  SPOs  via  a  Program 
Management  Directive  (PMD)  regarding  the  critical  test  and 
evaluation  issues  and  the  agencies  involved  (6:23). 

Sometimes  the  RTO  is  specifically  designated  in  the 
directive.  Once  the  RTO  for  the  test  program  is  identified, 
initial  planning  commences.  Planning  is  done  via  the  Test 
Planning  Working  Group  (TPWG)  as  directed  by  AFR  80-14. 
Members  of  this  planning  group  include  the  SPO  test  manager 
as  chairman,  and  representatives  from  the  RTO  (i.e.,  the 
test  agency),  the  OT&E  command,  operating  commands, 
supporting  commands,  par  t  i  c  i  pa  *•  ing  commands,  and  contractors 
(  7  :  5  ^  . 

Several  documents  regarding  the  test  and  evaluation  of 
the  program  are  generated  by  the  SPO  to  facilitate  the 
planning  and  implementation  of  test  management.  One  such 
document  is  the  Test  and  Evaluation  Master  Plan  (TEMP).  As 
described  in  AFR  80-14,  the  Test  and  Evaluation  Master  Plan 
will  do  the  following: 

.  .  .  integrate  critical  issues,  test  objectives, 

evaluation  criteria,  system  characteristics, 
responsibilities,  resources,  and  schedules  for  [Test 
and  Evaluation].  (7:5) 
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Another  document,  the  Program  Introduction  document, 
more  directly  affects  the  way  in  which  the  RTO  will  report 
to  the  SPO.  The  Aeronautical  Systems  Division  (ASD)  guide 
for  test  and  evaluation  management  states  that  "The  [Program 
Introduction  document]  is  the  primary  document  used  to 
request  AF  test  center/organizat i on  support  of  [Test  and 
Evaluation]"  (6:98).  The  guide  also  states  that  the 
Production  Introduction  document  should  "...  include 
requirements  for  fiscal  and  management  information  data, 
[and]  technical  reports  .  .  ."  (6:98). 

This  Program  Introduction  document  is  the  basic 
document  in  which  a  SPO  test  manager  formally  communicates 
hi?  cr  her  perceived  test  information  needs  to  the  RTO. 
Negotiations  between  the  SPO  and  RTO  over  the  information 
requirements  occur  during  the  preparation  of  this  important 
document.  Sometimes  the  RTO  is  inflexible  (for  various 
reasons)  in  its  negotiations  over  the  perceived  information 
needs  of  the  SPO  test  manager.  An  important  part  of  this 
research  is  to  identify  similarities  and  differences  of  SPO 
managers  and  RTO  managers  in  their  perceptions  of  what 
information  is  negotiated  and  their  perceptions  of  how 
flexible  RTO  managers  are  in  information  negotiations. 

Once  program  requirements  and  constraints  are 
communicated  to  the  RTO  by  the  SPO  via  the  Program 
Introduction  document,  the  RTO  will  communicate  its 
understanding  of  those  requirements  and  contraints  back  to 


the  SPO.  This  response  by  the  RTO  takes  the  official  form 
of  a  Statement  of  Capability.  The  Statement  of  Capability 
defines  the  level  of  support  to  be  provided  by  the  RTO  and 
specifies  the  funding  reimbursement  estimate  required  of  the 
SPO  (6:99).  The  Program  Introduction  document  and  the 
Statement  of  Capability  will  together  identify  the  agreement 
between  the  SPO  and  the  RTO  on  the  reporting  requirements 
(among  other  test  and  evaluation  requirements)  that  are 
necessary  to  manage  the  test  program. 

If  reporting  requirements  are  not  clearly  specified  in 
the  Program  Introduction  document-statement  of  Capability 
agreement,  the  possibility  exists  that  the  information  may 
not  be  reported,  or  if  reported,  the  information  may  not  be 
in  the  form,  frequency,  or  manner  desired  by  the  SPO.  SPOs 
will  continually  negotiate  with  RTOs  over  the  reporting 
requirements  of  the  test  program  (i.e.,  the  formal  written 
negotiations  are  not  the  only  negotiations  that  occur).  In 
addition,  RTOs  have  guidelines  and  policies  that  govern  what 
and  how  they  report  test  progress  information  to  their 
customers,  and  they  usually  negotiate  from  that  basis. 

Test  Reporting  Policy 

A  literature  review  of  the  policies  and  regulations 
that  govern  the  A950th  Test  Wing  reporting  of  test  program 
progress  to  their  customers  revealed  what  type  of 
information  is  generally  reported.  The  A950th  Test  Wing 
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regulation,  4950  TESTWR  80-4,  states  the  following  about 

reporting  progress  to  the  customer: 

Progress  reportCing]  will  normally  include  a  short 
narrative  which  will:  (1)  provide  an  analysis  of 
cost/schedule  growth  or  problems,  (2)  provide  other 
general  information  to  the  customer  about  the  status  of 
the  test  item,  aircraft,  test  range,  etcetera  and,  (3) 
provide  the  customer  with  the  "flavor"  of  the  test 
progress  .  (9:21) 

The  inherent  flexibility  of  this  policy  gives  the  Test 
Wing  latitude  as  to  how  it  can  report  to  the  SPO,  given 
manpower  and  funding  constraints. 

Another,  governing  document,  ASDP  80-14,  describes  what 
other  test  wing  responsibilities  test  reporting  also 
i nvo Ives: 

.  .  .  to  ensure  maximum  use  of  the  information 

generated  as  a  result  of  the  test,  the  reports  must  be 
timely,  factual,  concise,  and  complete.  The  reports 
should  be  balanced  and  accurately  present  an  overall 
portrayal  of  the  system  under  test.  (6:44) 

The  determination  of  whether  or  not  reporting  is 

timely,  factual,  concise,  complete,  balanced,  and  accurate 

can  be  judged  by  the  satifaction  of  the  customer  receiving 

the  reported  information.  This  research  will  (among  other 

things)  identify  customer's  perception  of  how  well  the  Test 

Wing  test  managers  have  done  overall  in  reporting  information. 


III.  Methodology 


Introduct i on 

The  focus  of  this  research  methodology  was  to  provide 
the  logical  mechanism  in  which  to  meet  the  research 
objectives.  A  study  of  the  4950th  Test  Wing  was  conducted 
to  assess  perceptions  of  and  performance  in  reporting  flight 
test  information  to  ASD  SPOs  and  WRDC  Labs.  The  means  of 
collecting  data  on  the  Test  Wing’s  reporting  performance  as 
perceived  by  its  customers  was  done  by  interviewing 
individual  SPO/Lab  test  managers.  In  some  cases,  SPO/Lab 
program  managers  were  interviewed  because  they  were 
considered  the  counterpart  manager  to  whom  the  Test  Wing 
manager  reported  information.  Hereafter,  any  reference  to 
SPO/Lab  test  managers  will  include  the  SPO/Lab  program 
managers  because  of  their  involvement  in  testing.  Test  Wing 
test  managers  were  also  interviewed  in  the  same  manner  as 
the  SPO/Lab  managers  to  determine  their  perceptions  about 
their  own  performance  in  reporting  information  to  their 
customers . 

Populat ion 

The  population  of  test  managers  targeted  for  interview 
was  derived  from  the  total  population  of  all  4950th  Tast 
Wing  customer'’..  Test  Wing  clientel  include  such  customers 
as  local  ASD  and  WRDC  program  offices,  Electronic  Systems 
Division  (ESD)  program  offices,  operational  command  (Hq  MAC) 
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offices,  U.S.  Army  and  Navy  organizations,  and  other  test 
wings.  Due  to  constiaints  on  time  and  funding,  interviews 
could  not  be  conducted  with  all  Test  Wing  customers.  Also, 
the  differences  in  customer  missions  and  locations  would 
significantly  impact  the  way  in  which  the  Test  Wing  reports 
information  to  its  customers.  Consequently,  this  research 
was  limited  to  local  ASD  and  WRDC  program  offices. 

Lists  of  ASD  and  WRDC  flight  test  programs  were 
obtained  from  the  4950th  Test  Wing  Directorate  of  Resource 
Management.  The  lists  included  all  program?  in  either  the 
testing,  final  reporting,  or  recently  completed  phase. 
Programs  in  the  planning  phase  were  omitted  because  there 
would  have  been  little  or  no  reporting  of  information  to  the 
customer.  Programs  that  were  completed  earlier  than  four 
months  prior  to  the  date  of  the  research  were  not  considered 
due  to  the  changeover  of  personnel  and  loss  of  recollection 
of  program  details. 

ASD  and  WRDC  programs  are  included  together  in  the 

population  of  interest,  even  though  they  differ  from  each 

other  in  the  nature  of  their  business.  ASD  flight  test 

programs  are  oriented  toward  development  tosting-- 

specifically,  testing  for  the  following  reasons; 

.  .  .  [to]  demonstrate  for  the  Air  Force  that  the 

system  engineering  design  and  development  is  complete, 
that  design  risks  have  been  minimized,  and  that  the 
system  will  perform  as  required  in  its  intended 
environment.  (6:2) 
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On  the  other  hand,  WRDC  flight  test  programs  are  oriented 
more  toward  research  testing  (i.e.,  proving  the  feasibility 
of  technologies ) . 

The  population  of  test  managers  is  defined  as  those 
test  managers  who  are  directly  responsible  for  the  daily 
management  of  the  test  program,  either  from  the  Program 
Office  side  or  the  Test  Wing  side.  The  arrangement  is  that 
a  specific  program  has  two  responsible  test  managers  (the 
ASD/WRDC  test  manager  and  the  Test  Wing  test  manager)  that 
interface  with  each  other  on  the  day-to-day  flight  test 
activities.  The  Test  Wing  test  manager  reports  test  program 
progress  information  to  the  ASD/WRDC  test  manager. 

Ideally,  these  counterpart  managers  would  be  test  managers 
who  have  worked  on  the  program  since  its  inception  and 
served  as  the  full  time  focal  point  throughout  testing  and 
reporting.  Because  of  a  high  rate  of  turnover  for 
management  personnel,  the  ideal  situation  cannot  always  be 
achieved;  however,  interviews  were  conducted  with  those 
managers  that  most  closely  fit  the  ideal  case. 

Data  Collection 

Data  was  collected  via  interviews  with  test  managers 
and  program  managers  trom  ASD  and  WRDC,  and  with  the 
managers'  counterparts  in  the  A950th  Test  Wing.  These 
managers  were  associated  with  nine  flight  test  programs. 

The  total  number  of  managers  interviewed  was  10.  Of  the 
initial  21  programs  that  were  possible  candidates  for 
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research,  only  nine  met  the  research  r equ i rement s - -s i x  ASD 
programs  and  three  WRDC  programs.  Reasons  for 
disqualification  of  these  and  the  other  unsuitable  programs 
are  the  following:  program  termination;  program 
sensitivity;  programs  still  remain  in  the  planning  phase  and 
have  not  yet  entered  flight  testing  and  information 
reporting;  no  knowledgable  counterpart  test  managers 
remaining/available  for  interview  (i.e.,  the  test  managers 
had  retired);  the  Test  Wing  was  not  serving  the  SPO/Lab  as 
the  RTO  (i.e.,  the  Test  Wing  was  just  providing  airframe  or 
maintenance  support  to  the  program);  and  programs  were  new 
thrust  programs  unique  to  the  Test  Wing  (i.e., 
manufacturing  support  programs  with  no  flight  testing 
involving  the  SPOs  or  Labs).  Before  the  18  interviews  from 
the  nine  programs  were  completed,  an  updated  list  from  the 
4950th  Test  Wing  became  available.  An  additional  eight 
possible  candidate  programs  were  identified;  however,  in 
telephone  conversations  with  the  respective  Test  Wing  test 
manager.s,  all  eight  were  found  not  suitable. 

Each  of  the  10  interviews  was  conducted  individually 
and  in  the  same  format  and  manner  to  avoid  bias  and  thus 
maintain  integrity  of  data.  A  script  of  the  interview 
questions  is  provided  in  Appendix  A.  Anonymity  was  assured 
to  test  managers  being  interviewed,  and  they  were  informed 
that  their  responses  would  be  associated  with  generic 
program  titles  labelled  by  alphabet  rather  than  by  name. 


The  interviews  involved  collecting  data  in  four  areas 
to  answer  the  investigative  questions  posed  in  Chapter  I. 

The  first  line  of  interview  questions  solicited  a 
definitive  list  of  categories  in  which  Test  Wing  managers 
report  test  program  information  to  their  customers  before, 
during,  and  after  fl’ght  test  events  occur.  In  a  two-step 
process,  each  counterpart  test  manager  (Test  Wing  and 
SPO/Lab)  was  asked  to  identify  those  categories  they  felt 
were  applicable  to  their  test  program.  First,  the  managers 
drew  a  line  through  non-appl icable  categories  on  a 
prescribed  list.  The  prescribed  list  was  taken  from  the 
4950th  Test  Wing’s  Test  Director’s  Guide,  4950  TESTWP  80-1, 
Appendix  H  (2:H-2).  Descriptions  of  each  category  on  the 
prescribed  list  were  not  published  in  this  Test  Wing 
pamphlet;  however,  descriptions  were  derived  from  a 
discussion  with  the  4950th  Test  Wing’s  point  of  contact 
concerning  the  pamphlet's  Appendix  H  and  are  presented  in 
Appendix  8  (15).  During  the  interview,  each  respondent  was 
encouraged  to  ask  for  clarification  of  category  meanings  to 
ensure  understanding  of  the  categories  and  consistency 
between  respondents.  In  the  second  step  of  the  process  of 
identifying  categories,  the  managers  added  any  categories 
they  felt  were  applicable  to  their  test  program  but  were  not 
on  the  prescribed  list. 

Once  the  list  was  defined,  the  test  manager  was  asked 
to  assign  relative  weights  of  importance  to  each  of  the 
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categories.  A  Likert  scale  was  presented  to  the  interview 
respondent  (as  shown  in  Figure  1)  for  him  or  her  to  use  in 
rating  the  perceived  importance  of  each  category  of 
inf ormat i on . 


Likert  Scale  1:  Importance  of  Categories 

"How  important  do  you  consider  this  category  to  be 
(independent  from  the  others)  in  reporting  information?" 


LESS 

VERY 

IMPORTANT 

IMPORTANT 

1 

2 

3 

A 

5 

Figure  1.  Information  Importance  Interview  Question  and 
Likert  Scale 

The  oecond  set  of  questions  posed  to  test  managers  was 
to  determine  perceptions  about  Test  Wing  managers’ 
flexibility  in  negotiating  what  and  how  information  was  to 
be  reported  to  meet  SPO/Lab  test  information  needs.  The 
test  manager  was  asked  to  determine  a  list  of  categories  of 
information  of  which  there  had  been  an  occasion  to 
negotiate  over  what  and  how  information  was  to  be  reported. 
Then  the  manager  was  asked  to  rate  via  a  Likert  scale  (see 
Figure  2)  how  flexible  in  negotiations  the  Test  Wing  was 
perceived  to  be  in  each  category. 
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Likert  Scale  2:  Reporting  Negotiability  (Test  Wing 
vers i on ) 

"How  flexible  in  reporting  negotiations  do  you  feel 
you  were  in  meeting  your  customer's  perceived  information 
needs  for  this  category?" 

NOT  NEGOTIABLE  EXTREMELY 

AT  ALL  NEGOTIABLE 

12  3  4  5 


Likert  Scale  2:  Reporting  Negotiability  (SPO/Lab  version) 

"How  flexible  in  reporting  negotiations  do  you  feel 
the  Test  Wing  was  in  meeting  your  perceived  information 
needs  for  this  category?" 

NOT  NEGOTIABLE  EXTREMELY 

AT  ALL  NEGOTIABLE 

1  2  3  4  5 


Figure  2.  Reporting  Negotiability  Interview  Question  and 
Likert  Scale 

The  third  area  of  interview  questions  sought  to  collect 
data  on  the  Test  Wing's  overall  performance  with  respect  to 
its  reporting  of  test  progress  information.  The  objectives 
were  to  determine  in  what  categories  information  had 
actually  been  reported  and  to  determine  how  well  the  Test 
Wing  was  perceived  to  have  done  in  reporting  inf ormat i on- -by 
its  customers  and  in  its  own  perception.  Once  again  a 
Likert  scale  was  used  (Figure  3)  to  rate  perceptions  of  how 
well  the  Test  Wing  performed  overall  in  reporting 
informat  ion . 
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Likert  Scale  3:  Overall  Reporting  Performance  (Test  Wing 
version) 


"How  well  do  you  feel  you've  done  overall  in 
reporting  information  for  this  category  to  your  customer?" 

POOR  OUTSTANDING 

12  3  4  5 

Likert  Scale  3:  Ovej.ali  Reporting  rerformonce  (SFG/LaL 
vers  ion ) 

"How  well  do  you  feel  the  Test  Wing  has  done  overall 


in  reporting 

information  to  you  for 

this 

category? " 

POOR 

OUTSTANDING 

1 

2  3 

4 

5 

Figure  3.  Overall  Reporting  Performance  Interview  Question 
and  Likert  Scale 

The  fourth  area  of  interest  was  to  determine  the  method 
that  was  most  commonly  used  in  reporting  information  for 
each  category.  Primary  and  other  communication  methods  were 
recorded  along  with  comments  about  efficiency  and 
effectiveness  of  the  methods.  Interview  responses  for  this 
fourth  area  are  recorded  in  Table  11  of  Chapter  IV. 

Data  Analysis 

The  methodology  used  for  analyzing  the  data  was  one 
that  assessed  the  four  areas  of  reporting  information 
discussed  in  the  previous  section  (i.e.,  information 
importance,  reporting  negotiability,  overall  reporting 
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performance,  and  methods  of  communication).  The  interview 
responses  for  the  first  three  areas  were  compared  and 
analyzed  using  descriptive  statistics  (mean  values)  and 
statistical  tests  of  significance.  The  fourth  area  under 
consideration,  methods  of  communication,  was  also  analyzed 
using  descriptive  statistics  (frequency  counts  instead  of 
mean  values).  Analysis  of  the  four  areas  mentioned  above 
was  followed  by  discussion  about  reporting  issues  that 
surfaced  in  the  open-ended  comments  made  by  the  interview 
respondents  (i.e.,  problems  encountered  in  reporting  or 
receiving  information  and  any  recommendations  for 
improvement ) . 

Descriptive  statistics  was  the  first  method  used  in 
analyzing  the  areas  of  information  importance,  reporting 
negotiability,  and  overall  reporting  performance.  The 
procedure  for  descriptive  statistical  analysis  was  to  first 
pair  the  responses  from  all  of  the  interviews  for  each 
category  of  information.  Then  mean  values  were  computed 
from  the  Likert  scale  rankings  for  each  category  of 
information  within  each  of  the  three  areas  mentioned  above. 
These  mean  values  were  tabulated  and  plotted  for  comparison 
between  Test  Wing  manager  responses  and  SPO/Lab  manager 
responses .  Ranges  of  the  Likert  scale  responses  were  also 
determined,  tablulated,  and  compared  for  each  category 
within  each  of  the  three  areas. 
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The  second  method  of  data  analysis  used  for  the  areas 

of  information  importance,  reporting  negotiability,  and 

overall  reporting  performance  was  the  statistical  test  of 

significance.  This  type  of  analysis  was  made  using  the 

statistical  analysis  program  for  microcomputers  called 

STATISTIX.  STATISTIX  contains  the  capability  to  perform  the 

two-sample  parametric  t  test  or  a  nonpar ametr i c  alternative 

test  called  the  rank  sum  test  (also  known  as  the  Wilcoxon 

rank  sum  test).  STATISTIX  documentation  describes  the  two- 

sample  t  test  in  the  following: 

This  procedure  computes  two  sample  t  tests,  which  test 
for  differences  between  the  means  of  two  independent 
samples.  It  is  applicable  to  situations  where  samples 
are  drawn  independently  from  two  normally  distributed 
groups .  (11:6.10) 

The  two-sample  t  test  is  not  an  appropriate  statistical  test 
if  the  observations  from  each  population  are  not  normally 
distributed. 

The  nonpar ametr i c  alternative  to  the  two-sample  t  test 
(i.e.,  the  rank  sum  test)  accomodates  less  stringent 
requirements  for  the  response  data  to  be  normally 
distributed  and  for  the  variances  to  be  equal.  STATISTIX 
describes  the  rank  sum  test  as  a  test  for  differences  in  the 
central  values  of  samples  (populations  in  this  case)  from 
two  independent  samples  (populations)  (11:6.12). 

Before  discussing  the  procedures  used  in  performing  the 
statistical  tests  of  significance  (parametric  and 
nonparametr i c )  using  the  STATISTIX  analysis  program,  it  is 
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appropriate  to  address  the  following  assumptions  for 
parametric  testing  of  interview  data  (10:358;  14:174,187; 
12:111,113,118,151,152).  The  only  assumption  that  applies 
in  the  nonpararaetr i c  testing  is  the  first  assumption. 

1.  The  observations  are  independent.  Observe  *■  i  ons  are 
defined  here  to  be  the  Likert  scale  responses  of  each 
interviewea  manager  for  each  category  of  information.  The 
observations  are  independent  from  the  standpoint  that  the 
managers  of  the  test  programs  from  the  Test  Wing  side  and 
the  SPO/Lab  side  each  have  their  own  unique  perceptions 
about  information/reporting  requirements.  Test  Wing 
managers  have  the  perspective  of  managing  or  monitoring  the 
day-to-day  flight  test  activities  of  a  test  program.  The 
SPO/Lab  test  managers  have  the  perspective  of  a  Test  Wing 
customer  and  of  a  manager  of  the  overall  test  program 
effort.  Thus,  the  two  populations  of  means  are  considered 
independent.  The  interviews  were  conducted  in  such  a  way  as 
to  seek  independent  responses  from  these  managers  about 
their  own  perceptions.  These  managers  were  asked  to  refrain 
from  any  discussion  with  their  counterparts  about  this 
research  so  as  not  to  bias  their  independent  responses. 

2.  The  observations  should  be  drawn  from  normally 
distributed  populations.  Each  of  the  two  mean  value 
populations  that  were  derived  from  the  Likert  scale 
numerical  ratings  of  the  Test  Wing  managers  and  of  the 
SPO/Lab  managers  were  initially  considered  to  be  normally 
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distributed.  Verification  that  mean  values  are  or  are  not 
normally  distributed  is  demonstrated  via  a  Wilk- 
Shapiro/Ranki t  Plot  and  the  W  test  (where  W  denotes  the 
Wilk-Shapiro  test  statistic).  The  STATISTIX  documentation 
states  the  following  about  checking  the  normality 
as sump t ion : 

Wi Ik-Shapiro/Ranki t  Plots  (examine]  whether  a  variable 
conforms  to  a  normal  distribution  .  .  .  [and]  .  .  . 

departure  of  the  Rankit  plot  from  a  linear  trend 
indicates  non-normality,  as  does  a  small  value  for  the 
Wilk-Shapiro  statistic  [W].  (11:0.4-8.5) 

Rankit  plots  were  performed  on  the  data  for  all  categories, 

and  the  W  yielded  by  STATISTIX  output  was  compared  to 

quantiles  (p  values)  for  a  95%  confidence  level  to  determine 

normality  (16:605). 

3.  The  populations  of  interest  should  have  equal 
variances.  The  two-sample  t  test  of  the  STATISTIX  analysis 
program  performs  tests  for  unequal  variances  as  well  as  for 
equal  variances.  Computer  output  from  the  t  test  provides  F 
test  data  for  determination  of  equality  of  variances,  and  it 
also  provides  t  test  statistics  and  p  values  for  equal  and 
unequal  variances.  In  some  cases,  the  output  revealed  that 
it  was  highly  likely  that  the  populations  had  unequal 
variances.  For  those  cases,  the  p  value  associated  with  the 
unequal  variance  was  used  to  determine  if  there  was 
significant  difference  in  the  two  mean  values. 

4.  The  measurement  levei  of  data  should  be  at  least 
interval  level  so  that  arithmetic  operations  (computation  of 
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mean  values)  can  be  used.  It  was  possible  to  use  arithmetic 
mean  values  for  statistical  analysis  and  comparison  of 
interview  responses  because  the  measurement  level  of  Likert 
scale  response  data  is  presumed  to  be  interval  level  data. 

In  his  book,  Business  Research  Methods.  C.  William  Emory 
states  that  “Many  attitude  scales  are  presumed  to  be 
interval,  although  such  claims  are  often  challenged" 

(10:91).  Emory  further  states,  "Obviously  the  data  does  not 
know  what  scale  it  is,  so  whether  a  particular  scale  is 
interval  or  ordinal  often  is  a  matter  of  judgment"  (10:91). 

While  the  argument  may  exist  as  to  whether 
opinion/attitude  data  from  Likert  scale  rankings  is 
appropriately  considered  to  be  interval  or  ordinc 1 , 
inferential  statistical  tests  (parametric  tests  using  mean 
values)  have  been  and  are  performed  on  Likert  scale  data  as 
if  the  data  measurement  level  is  interval.  Emory  states 
that  ".  .  .  some  behavioral  scientists  argue  that  parametric 

tests  are  usually  acceptable  for  ordinal  scales  on  both 
practical  and  theoretical  grounds"  (10:90).  Parametric 
tests  (namely  the  t  test)  involving  the  comparison  of  mean 
values  are  considered  appropriate  for  this  analysis  because 
of  the  following: 

.  .  if  the  testing  situation  involves  two  independent 

samples  with  interval  or  ratio  measurements,  one  should 
use  the  t  test  of  differences.  (10:359) 

With  the  assumptions  now  defined,  it  is  appropriate  to 
discuss  the  procedures  used  in  the  statistical  analysis.  If 
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the  assumptions  described  above  were  met,  then  the  procedure 
for  analysis  would  be  one  for  the  two-sample  t  test.  If  the 
assumptions  were  not  met  (except  for  the  assumption  of 
independent  observations),  then  the  nonparametr ic  rank  sum 
test  procedure  was  followed.  The  rank  sum  test  is  a 
suitable  alternative  to  the  two-sample  t  test  because  it 
tests  for  the  significant  difference  between  two 
populations,  and  it  is  less  stringent  on  its  requirements 
for  assumptions.  The  rank  sum  test  does  require  that  the 
populations  be  independent  (as  described  in  the  first 
assumption).  In  addition  to  not  requiring  normally 
distributed  populations,  the  rank  sum  test  accomodates 
sample  sizes  (in  this  case,  population  sizes)  of  ten  or  less 
(14:184,187). 

The  idea  behind  the  statistical  analysis  using  the  two- 
sample  t  test  is  to  make  an  inference  as  to  whether  or  not 
the  mean  values  of  the  two  sample  responses  (i.e.,  the  two 
population  distributions  of  Likert  scale  responses)  differ 
significantly.  A  null  hypothesis  about  the  inference  is 
established.  The  null  hypothesis  is  that  there  is  no 
significant  difference  between  the  Test  Wing  mean  values  and 
the  SPO/Lab  mean  values  for  any  of  the  categories  of 
information.  In  other  words,  the  two  mean  values  for  any 
category  are  essentially  equal.  The  alternate  hypothesis 
is  that  there  is  a  significant  difference  between  the  two 
mean  values  for  any  category  of  information  (i.e.,  the  two 


mean  values  for  any  category  are  not  considered  equal).  The 
level  of  confidence  established  for  these  tests  is  95%,  a 
typical  standard.  A  decision  rule  is  established  to 
determine  when  the  null  hypothesis  is  to  be  rejected  as 
being  false  or  not  rejected  as  being  false.  Once  the  t  test 
statistic  is  computed  from  the  STATISTIX  analysis  program, 
the  corresponding  p  value  (also  yielded  by  the  STATISTIX 
analysis  program)  is  compared  to  the  95%  confidence  value  of 
0.05  (1.00  minus  0.95).  If  the  p  value  is  less  than  or 
equal  to  0.05,  then  the  decision  is  made  to  reject  the  null 
hypothesis  and  consider  the  two  mean  values  significantly 
different.  If  the  p  value  was  greater  than  0.05,  then  the 
decision  is  made  not  to  reject  the  ull  hypothesis  that  the 
mean  values  can  be  considered  equal.  A  tabulation  of  the  p 
values  and  test  results  along  with  a  discussion  about  the 
tabulated  results  is  presented  in  Chapter  IV. 

The  procedure  for  the  nonparametr i c  tests  of 
significance  is  similar  to  that  of  the  parametric  two-sample 
t  test;  however,  the  inference  about  the  populations  of 
observations  is  slightly  different.  The  null  hypothesis 
about  the  two  populations  of  Likert  scale  response  data 
(Test  Wing  and  SPO/Lab)  for  the  rank  sum  test  is  that  the 
two  populations  are  identical.  Compared  to  the  parametric 
two-sample  t  test,  the  rank  sum  test  determines  if  there  is 
a  significant  difference  between  the  central  values  of  two 
independent  populations,  rather  than  determining  if  there  is 
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a  significant  difference  in  mean  values  for  the  data 
(11:6.12).  The  alternate  hypothesis  for  the  rank  sum  test 
is  that  the  two  populations  are  not  identical.  Again  the 
level  of  confidence  is  set  at  95%.  The  decision  rule  for 
rejecting  the  null  hypothesis  is  established  ac  a  p  value  of 
0.05.  If  the  p  value  was  greater  than  0.05,  the  decision 
was  made  not  to  reject  the  null  hypothesis  that  the  two 
population  are  identical.  The  p  values  and  test  results  are 
presented  in  place  of  the  tabulation  of  two-sample  t  test 
results  if  the  assumption  of  normality  is  violated. 

The  last  of  the  four  areas  to  be  analyzed  was  the 
methods  Test  Wing  managers  use  to  communicate  to  their 
counterparts.  The  most  commonly  used  method  of 
communication  and  any  other  methods  identified  by  the 
managers  were  divided  into  classes.  Frequency  counts  of  the 
number  of  methods  in  each  class  were  made  for  the  primary 
methods  of  communication  identified  by  each  manager. 
Comparisons  were  made  of  the  frequencies  at  which  the 
methods  were  identified  by  the  Test  Wing  and  SPO/Lab 
managers  for  each  category. 

Following  analysis  of  the  four  areas  mentioned  above, 
reporting  issues  from  the  interview  respondents'  open-ended 
comments  were  discussed.  Comments  were  solicited  to 
identify  differences  in  reporting  practices  S PO  managers 
would  recommend.  Also,  comments  were  solicited  to  identify 
differences  Test  Wing  managers  would  prefer  in  reporting 
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information  to  their  customers.  The  discussion  concluded 
with  answers  to  the  research  questions  posed  in  Chapter  I. 
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IV.  Results  and  Analysis 


Introduct ion 

The  results  of  the  data  collection  will  be  presented 
and  analyzed  or  examined  for  the  following  four  areas: 
information  importance,  reporting  negot i abl i 1 i ty ,  overall 
reporting  performance,  and  methods  of  communication.  These 
four  areas  will  be  discussed  in  detail  in  the  sections  that 
follow.  Also  discussed  will  be  reporting  issues  from  the 
interview  respondent's  open-ended  comments.  The  discussion 
will  conclude  with  answers  to  the  research  questions  from 
Chapter  I.  First,  however,  will  be  a  presentation  and 
discussion  of  the  demographics  concerning  the  interview 
respondents  from  the  A950th  Test  Wing  and  from  the 
Aeronautical  Systems  Division  (ASD)  System  Program  Offices 
( S POs ) /Wr i gh t  Research  and  Development  Center  (WRDC) 
Laboratories  (Labs). 

Demographics 

Information  about  the  respondents  is  summarized  below. 
The  interview  demographic  questions  (see  Appendix  A) 
provided  information  about  the  respondents*  experience  in 
test  management  (in  two  cases,  program  management  as  it 
related  to  the  test  program),  their  experience  with  the 
program  of  interest,  the  current  phase/general  status  of  the 
program,  the  respondent's  counterpart  test/program  manager, 
and  the  approximate  number  of  times  per  week  the  two 
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counterpart  managers  were  in  contact  concerning  the  test 


program . 

Ranks/Grades  of  the  Respondents 


RANK/GRADE 

TW  RES. 

SPO/LAB  RES . 

Ma  jor 

- 

1 

1 

Captain 

- 

2 

1 

GM-14 

- 

- 

4 

GS-13 

- 

3 

3 

GS-12 

- 

2 

- 

CS-9 

- 

i 

— 

TOTAL 

9 

9 

Organ i za  t i on 

For  the  4950th  Test  Wing: 


OFFICE  SYMBOL  RES . 


AMX  -  1 

DOCA  -  1 

FFCA  -  1 

FFCE  -  2 

FFDA  -  1 

FFDS  -  3 

TOTAL  9 

For  the  SPOs/Labs: 

OFFICE  SYMBOL  RES ■ 


ASD/SDBG  -  1 
ASD/SDBX  -  1 
ASD/SDCT  -  3 
ASD/YWSB  -  1 
WRDC/AAAI  -  I 
WRDC/AAWD-l  -  2 

TOTAL  9 
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TW  RES. 


SPO/LAB  RES. 


Least  years 

1.5 

2 

TW  RES. 

SPO/LAB  RES. 

Most  years 

- 

6 

29 

MODE 

_ 

3.5 

1 1  and  14  ( Bimodal ) 

Years  Managing  the 


Least  years 
Most  years 

MODE 


Program  of  Interest 

TW  RES. 

9  months 

3.5 

2.5 


SPO/LAB  RES. 
1 

29 

6 


Paricipants  in  the  Original  Program  Introduction 
Document  ( PIP ) /Statement  of  Capability  (SC)  Negotiations 


For  the  A950th  Test  Wing: 

PARTICIPANTS 


Yes 

No 

For  the  SPOs/Labs: 


PARTICIPANTS 


Yes 

No 


RES  . 

5 

k 


RES  . 

7 

2 


Current  Phase  of  the  Test  Program 

The  following  table  shows  test  manager  reponses  when 


asked  what  phase  the  test  program  was  currently  in.  The 
discrepancy  in  responses  for  final  reporting  is  due  to  the 
time  lapse  between  interviews  of  counterpart  managers. 


TW  RES. _ SPO/LAB  RES. 


In  Testing  -  3  3 
In  Final  Reporting  -  2  0 
Completed  -  A  6 


Approximate  Number  of  Contacts  Made  per  Week  with 
Counterpart  Manager 

PROGRAM  TW  RES.  SPO/LAB  RES. 


A 

B 

C 

D 

E 

F 

G 

H 

I 


A 

2 

2 

5 

10-15 

2- 3 

3- A 
10 

1 


5 

3-A 

2-3/8-10 

5 

15 

1 

2 

5 


1-2 


★ 


*  Two  to  three  contacts  were  made  per  week:  when  there 
was  less  test  program  activity,  and  eight  to  ten  contacts 
per  week  were  made  during  active  testing. 


Information  Importance 

The  first  of  the  four  areas  on  which  the  interviews 
focused  is  the  area  of  information  category  importance.  The 
importance  area  entailed  identifying  the  categories  of 
information  that  the  test  manager  believed  to  be  relevant  to 
his  or  her  test  program.  In  a  two-step  process  described  in 
Chapter  III,  each  counterpart  test  manager  (Test  Wing  and 
SPO/Lab)  was  asked  to  draw  a  line  through  non-appl i cable 
categories  on  a  prescribed  list.  The  prescribed  list  of 
categories  was  taken  from  the  A950th  Test  Wing's  Test 
Director's  Guide.  A950  TESTWP  80-1,  Appendix  H  (80-1). 
(Categories  from  the  list  and  their  descriptions  are 
presented  in  Appendix  B.)  During  the  interview,  each 
respondent  was  encouraged  to  ask  for  clarification  of 
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category  meanings  to  ensure  understanding  of  the  categories 
and  consistency  between  respondents.  Test  managers 
considered  most  of  the  categories  of  information  from  the 
list  to  be  directly  relevant  to  their  test  program.  In  the 
second  step  of  the  process  of  identifying  categories,  the 
managers  added  any  categories  they  felt  were  applicable  to 
their  test  program  but  were  not  on  the  prescribed  list. 

There  were  some  categories  for  which  the  interview 
responses  between  counterpart  managers  for  a  specific 
program  did  not  match.  Table  1  shows  a  listing  of  programs 
(generically  labelled  by  alphabet  rather  than  name  for 
anonymity),  and  the  categories  of  information  for  which 
there  was  no  match  between  the  counterpart  manager's 
perceptions  about  the  applicability  of  the  category  to  the 
test  program. 

As  noted  in  Table  1,  some  additional  categories  were 
considered  important  by  the  managers  being  interviewed  and 
were  added  to  the  list.  None  of  the  categories  added  by  a 
manager  matched  the  responses  of  his  or  her  counterpart 
manager . 

Reasons  that  the  managers  did  not  perceive  all  the  same 
categories  to  be  directly  relevant  to  their  test  program 
could  be  the  following:  different  perceptions  about  what 
the  category  of  information  entailed  (although  care  was 
taken  during  the  interviews  to  convey  the  same  description 
of  each  category  as  shown  in  Appendix  B);  varied  backgrounds 
and  experiences  of  each  manager;  varied  management 
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TABLE  1 

MISMATCHES  BETWEEN  COUNTERPART  TEST  MANAGERS’ 
PERCEIVED  CATEGORY  APPLICABILITY 
(INFORMATION  IMPORTANCE) 

MISMATCHED  CATEGORIES 

RANGE  SUPPORT 

SUB-CONTRACTOR  INTERFACE  *;  FINAL  REPORTING  * 

PROCUREMENT/SUPPLY;  ENGINEERING;  RANGE 
SUPPORT;  FLIGHT  PROFILES  ★ 

LOGISTICS;  TECHNICAL  PERFORMANCE;  RANGE 
SUPPORT 

PROCUREMENT/SUPPLY;  LOGISTICS;  AIRCRAFT 
MAINTENANCE;  RANGE  SUPPORT;  DEMODIFICATION; 
CHASE  PLANE  *;  TECH  PHOTO  * 

DEMODIFICATION  REQUIRED  BY  EXTERNAL  FACTORS  ★ 

CONTRACTOR  FLIGHT  TRAINING  *;  DEMODIFICATION 
REQUIRED  BY  EXTERNAL  FACTORS  * 

PROCUREMENT/SUPPLY;  MODIFICATION  DESIGN; 
COMBINED  TEST  TEAM  INTERFACE  * 

RANGE  SUPPORT;  PREFAB;  FAA  INTERFACE 
(SIMULATOR  CERTIFICATION)  * 

*  Responses  that  were  added  by  the  interview  respondent. 


perspectives  due  to  the  nature  of  each  manager's  perceived 
test  management  role  (in  two  cases,  the  Test  Wing's 
specified  counterpart  SPO  manager  was  a  program  manager,  not 
a  test  manager);  varied  perspectives  about  what  the 
information  requirements  of  the  test  program  are  or  would 
be;  and  varied  perspectives  about  what  information  the 
contractor  was  perceived  to  report  to  the  SPO/Lab  manager 
versus  what  the  Test  Wing  manager  was  expected  to  report 


PROGRAM 

A 

B 

C 

D 

E 

F 

G 

H 

I 
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(i.e.,  if  the  program  involved  contractor  managed  testing). 

In  the  cases  where  the  contractor  served  as  the  testing 
agency,  the  4950th  Test  Wing  served  as  the  SPO/Lab  official 
monitor  of  the  testing  activities.  The  Test  Wing  reported 
contractor  progress  and  problems  to  the  SPO/Lab  managers  via 
categories  of  information. 

Once  the  applicable  categories  were  identified  by  the 
respondent,  a  level  of  "importance"  was  placed  on  each 
category  by  assigning  a  numbered  rating  from  a  Likert  scale; 
where  one  equalled  little  importance  and  five  equalled 
strong  importance.  These  importance  ratings  were  compiled 
from  all  18  responses  and  averaged  for  each  category  to 
produce  a  mean  value.  Table  2  shows  the  mean  values  for 
each  category.  Beside  each  mean  value  is  the  percentage  of 
responses  for  that  category.  Also  included  in  the 
tabulation  is  the  range  (minimum  and  maximum)  of  the  Likert 
scale  ratings  for  each  respective  category. 

A  graph  comparing  the  mean  values  from  the  Test  Wing 
responses  and  the  SPO/Lab  responses  is  presented  in  Figure  4. 

In  analyzing  the  presentation  of  data  in  Table  2  and 
Figure  4,  one  can  see  from  the  response  percentages  that 
100%  of  the  Test  Wing  and  SPO/Lab  managers  considered  the 
following  13  categories  to  be  directly  relevant  in  reporting 
information  on  their  respective  programs:  funding, 
manpower,  documentation,  schedule,  planning,  safety,  cost, 
modification  installation,  test  item,  instrumentation, 
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TABLE  2 


INFORMATION  IMPORTANCE  LIKERT  SCALE  MEAN  VALUES, 
PERCENT  RESPONSES,  AND  LIKERT  SCALE  RESPONSE  RANGES 
FOR  THE  TEST  WING  AND  SPO/LAB 


TEST  WING  SPO/LAB 


CATEGORY 

MEAN 

%  RES. 

RANGE 

MEAN 

%  RES  . 

RANGE 

PROC/SUPPLY 

1  .  9 

78 

1 

TO 

3 

3  .  1 

89 

1 

TO 

5 

LOGISTICS 

2 . 9 

89 

1 

TO 

5 

3  .  3 

89 

1 

TO 

5 

FUNDING 

4.4 

100 

3 

TO 

5 

4  .  3 

100 

3 

TO 

5 

MANPOWER 

2 . 7 

100 

1 

TO 

4 

3 . 8 

100 

2 

TO 

5 

DOCUMENTATION 

3  .  1 

100 

2 

TO 

5 

3 . 8 

100 

3 

TO 

5 

SCHEDULE 

4 . 9 

100 

4 

TO 

5 

4 . 9 

100 

4 

TO 

5 

PLANNING 

4 . 7 

100 

4 

TO 

5 

4 . 2 

100 

2 

TO 

5 

ENGINEERING 

3 . 9 

100 

3 

TO 

5 

3 . 8 

89 

2 

TO 

5 

SAFETY 

3 . 6 

100 

2 

TO 

5 

4 . 2 

100 

2 

TO 

5 

MOD  DESIGN 

3  .  1 

89 

2 

TO 

5 

3 . 8 

100 

2 

TO 

5 

COST 

4 . 2 

100 

2 

TO 

5 

4 . 0 

100 

2 

TO 

5 

MOD  INSTAL'N 

3  .  7 

100 

3 

TO 

5 

3 . 8 

100 

2 

TO 

5 

A/C  MAINT 

2.8 

89 

1 

TO 

5 

3.2 

100 

2 

TO 

5 

TEST  ITEM 

4  .  1 

100 

3 

TO 

5 

3 . 6 

100 

2 

TO 

5 

INSTRUMENT ' N 

3 . 9 

100 

3 

TO 

5 

3 . 9 

100 

3 

TO 

5 

FLT  TESTING 

5 . 0 

100 

NONE 

4 . 8 

100 

4 

TO 

5 

TECH  PERFORM 

4 . 8 

89 

4 

TO 

5 

4 . 0 

100 

3 

TO 

5 

RANGE  SPT 

3.  3 

67 

1 

TO 

4 

4  .  3 

78 

3 

TO 

5 

DATA  ANALYSIS 

3.8 

100 

2 

TO 

5 

4 . 0 

100 

3 

TO 

5 

DEMOD 

3 . 0 

78 

2 

TO 

5 

2 . 6 

89 

1 

TO 

4 

CONTR'R  INT 

3 . 8 

100 

2 

TO 

5 

3 . 8 

100 

1 

TO 

5 

PREFAB 

2 . 8 

68 

2 

TO 

5 

3 . 0 

56 

1 

TO 

5 

flight  testing,  data  analysis,  and  contractor  interface.  In 
addition,  100%  of  the  Test  Wing  managers  felt  that 
engineering  information  was  directly  relevant  to  their  test 
program.  Likewise,  100%  of  the  SPO/Lab  managers  felt  that 

H 

modification  design,  aircraft  maintenance,  and  technical 
performance  were  directly  relevant  to  their  program. 

Another  observation  from  Table  2  is  that  there  are  no 
absolute  polarizations  of  any  Test  Wing  and  SPO/Lab  mean 
values.  The  most  that  two  paired  means  differ  is  by  1.2 
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CATEGORY  (5  CATEiGOPY 


COST 
MOO  DESGN 
SAFETY 
ETOCERNG 
PLATWNG 
SCHEDULE 
OOCUKENTATTON 

manpower 

PJNONG 

LOOSTCS 

PROC/'SUPPLY 


0,0  1,0  2.0  3.0  4.0  5.0 


TW 


SP0/LA8 


RATING 


jre  A.  Information  Importance  Mean  Comparison 


prefab 

COTTR  NT 

DEMOO 
lata  atjalys 
RaIKSE  spt 
TECHi  FERF 

flt  testing 

NSTRL».£NT’N 
TEST  ITEM 
A/'C  K.tA,fJT 
MOO  IT.'STALL'N 

0.0  1.0  2.0  3.0  4  0  5.0 

MEATJ  RATNG 


I 


Figure  A,  continued.  Information  Importance  Mean  Comparison 


39 


points  (for  the  category  of  Procurement/Supply ) .  Other 
categories  that  differ  by  one  point  or  more  are  manpower  and 
range  support.  There  are  three  categories  for  which  the 
mean  values  are  equal  for  the  Test  Wing  responses  and 
SPO/Lab  responses.  Those  categories  are  schedule, 
instrumentation,  and  contractor  interface.  For  16  of  the  22 
categories,  the  SPO/Lab  mean  values  are  greater  than  the 
Test  Wing  mean  values.  Finally,  the  category  in  which  the 
least  number  of  managers  (56%)  perceived  it  was  directly 
relevant  to  their  test  program  is  the  prefab  category. 

As  discussed  in  the  data  analysis  section  of  Chapter 
III,  arithmetic  mean  values  were  to  be  the  basis  of 
comparison  for  statistical  analysis.  Original  intentions 
were  that  a  parametric  analysis  be  performed  on  the  Test 
Wing  responses  and  the  SPO/Lab  responses  via  a  two-sample  t 
test.  The  parametric  analysis  was  to  be  performed  to 
determine  if  there  is  any  significant  statistical  difference 
between  the  Test  Wing  mean  values  and  the  SPO/Lab  mean 
values  for  each  category’s  Likert  scale  responses. 

Before  the  two-sample  t  tests  were  performed  on  the 
data,  the  necessary  assumptions  about  the  data  (see  Chapter 
III,  Data  Analysis  section  for  a  description  of  the 
assumptions)  were  checked  for  validity.  For  13  of  the  22 
information  categories,  the  Likert  scale  response  data  could 
not  be  assumed  to  be  normally  distributed.  (The  validity 
check  for  normality  is  also  described  in  Chapter  III.)  In 
order  to  continue  a  statistical  analysis  of  the  data. 
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nonparametr i c  tests  were  performed  using  the  rank  sum  test 
(a  variation  of  the  Wilcoxon  rank  sum  test).  The  results  of 
these  nonparametr i c  tests  are  found  in  Table  i. 

Table  3  contains  p  values  and  an  indication  of  whether 
or  not  the  populations  of  Likert  scale  responses  are 
i dent i ca 1 . 


TABLE  3 

RESULTS  FROM  THE  RANK  SUM  TESTS 
(INFORMATION  IMPORTANCE) 


IDENTICAL  POPULATION 


CATEGORY 

P  VALUE 

DISTRIBUTIONS? 

PROCUREMENT/SUPPLY 

0.08  * 

YES 

LOGISTICS 

0 .42 

YES 

FUNDING 

0.67  + 

YES 

MANPOWER 

0 .02 

NO 

DOCUMENTATION 

0 . 00 

YES 

SCHEDULE 

1.00  + 

YES 

PLANNING 

0.30  +  * 

YES 

ENGINEERING 

0  .  77 

YES 

SAFETY 

0.21  + 

YES 

MOD  DESIGN 

0.15  + 

YES 

COST 

0.85  + 

YES 

MOD  INSTALLATION 

0.76  + 

YES 

A/C  MAINTENANCE 

0  .  30 

YES 

TEST  ITEM 

0.15 

YES 

INSTRUMENTATION 

1.00  + 

YES 

FLIGHT  TESTING 

0.34  + 

YES 

TECH  PERFORMANCE 

0.05  + 

NO 

RANGE  SUPPORT 

0.07  + 

YES 

DATA  ANALYSIS 

0.68  + 

YES 

DEMODIFICATION 

0 . 48 

YES 

CONTRACTOR  INT 

0 . 85 

YES 

PREFAB 

0.53  + 

YES 

+  Non-normal  distributions 
*  Unequal  variances 


The  population  distributions  of  Likert  scale  responses  were 
determined  to  be  identical  or  not  by  computing  the  Wilcoxon 
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rank  sum  test  statistic,  W,  and  comparing  the  test 
statistic's  corresponding  p  value  to  the  95%  confidence 
value  of  0.05  (1.00  minus  0.95).  If  the  p  value  is  less 
than  or  equal  to  0.05,  then  the  decision  is  made  to  reject 
the  null  hypothesis  and  consider  the  populations  as 
significantly  different.  If  the  p  value  was  greater  than 
0.05,  then  the  decision  is  made  to  not  reject  the  null 
hypothesis  that  the  populations  can  be  considered  identical 

In  all  but  two  categories  of  information  (manpower  and 
technical  performance),  the  nonparametr i c  analysis  indicate 
that  the  population  of  Likert  scale  responses  for  the  Test 
Wing  responses  do  not  differ  significantly  from  the 
population  for  the  SPO/Lab  responses  (i.e.,  the  population 
distributions  can  be  considered  identical).  This  indicates 
that  for  most  of  the  categories  (all  but  two),  managers  on 
both  sides  of  a  program  have  very  similar  perceptions  as  to 
the  importance  of  each  these  categories  of  information  as 
they  relate  to  their  specific  test  program. 

Reporting  Negotiability 

The  next  area  to  be  discussed  is  Test  Wing  manager 
negotiability  over  reporting  information.  Negotiability  in 
reporting  information  has  to  do  with  the  flexibility 
demonstrated  by  the  Test  Wing  manager  in  negotiations  over 
reporting  information  in  such  a  way  that  satisfies  the 
customer’s  perceived  information  needs.  In  each  interview, 
the  manager  reviewed  the  same  list  of  categories  that  he  or 
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she  identified  as  applicable  in  reporting  test  program 
progress.  From  this  list,  the  manager  was  asked  to  cross 
out  any  categories  for  which  no  negotiations  about  reporting 
information  took  place.  The  remaining  categories  were  those 
for  which  some  type  of  negotiations  occured  (via  verbal  or 
written  form,  such  as  agreements  written  into  the  Program 
Introduction  and  Statement  of  Capability  documents).  Of 
the  remaining  categories,  there  were  some  for  which  the 
counterpart  managers  for  a  specific  program  did  not  agree 
that  negotiations  took  place.  Table  A  shows  a  listing  of 
programs  and  categories  for  which  there  was  no  match  between 
counterpart  managers  about  negotiations  of  reporting 
information  and  also  shows  categories  that  were  added  by  the 
respondents.  None  of  the  categories  added  by  a  respondent 
were  also  added  by  his  or  her  counterpart  test  manager. 

TABLE  4 

MISMATCHES  BETWEEN  COUNTERPART  TEST  MANAGERS' 
PERCEPTIONS  ABOUT  CATEGORIES  IN  WHICH 
REPORTING  NEGOTIATIONS  OCCURED 
(REPORTING  NEGOTIABILITY) 

PROGRAM  MISMATCHED  CATEGORIES 

A  PROCUREMENT/SUPPLY;  MANPOWER;  SAFETY;  COST; 

AIRCRAFT  MAINTENANCE;  TECHNICAL  PERFORMANCE 

B  MANPOWER;  MODIFICATION  DESIGN;  MODIFICATION 

INSTALLATION;  TEST  ITEM;  TECHNICAL 
PERFORMANCE;  SUB-CONTRACTOR  INTERFACE  * 
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[TABLE  CONTINUED] 

PROGRAM  MISMATCHED  CATEGORIES 

C  PROCUREMENT/SUPPLY;  MANPOWER; 

DOCUMENTATION;SCHEDULE;  ENGINEERING ; 
MODIFICATION  DESIGN;  MODIFICATION 
INSTALLATION;  AIRCRAFT  MAINTENANCE; 
INSTRUMENTATION:  RANGE  SUPPORT;  DATA 
ANALYSIS;  DEMODIFICATION;  FLIGHT  PROFILES  * 

D  PROCUREMENT/SUPPLY;  LOGISTICS;  DOCUMENTATION; 

SCHEDULE;  PLANNING;  ENGINEERING;  SAFETY; 
*MODIFICATION  DESIGN;  AIRCRAFT  MAINTENANCE; 
RANGE  SUPPORT;  DEMODIFICATION;  CONTRACTOR 
INTERFACE;  PREFAB 

E  M,.NPOWER;  DOCUMENTATION;  ENGINEERING;  COST; 

TEST  ITEM;  CONTRACTOR  INTERFACE;  CHASE  PLANE* 

F  LOGISTICS;  SAFETY;  PREFAB;  DEMODIFICATION 

REQUIRED  BY  EXTERNAL  FACTORS  * 

G  AIRCRAFT  MAINTENANCE;  CONTRACTOR  INTERFACE; 

DEMODIFICATION  REQUIRED  BY  EXTERNAL  FACTORS*; 
CONTRACTOR  FLIGHT  TRAINING  * 

H  PROCUREMENT/SUPPLY;  LOGISTICS;  MANPOWER; 

DOCUMENTATION;  SCHEDULE;  PLANNING; 
ENGINEERING;  MODIFICATION  DESIGN; 

MODIFICATION  INSTALLATION;  AIRCRAFT 
MAINTENANCE;  TEST  ITEM;  INSTRUMENTATION; 
FLIGHT  TESTING;  TECHNICAL  PERFORMANCE;  RANGE 
SUPPORT;  COMBINED  TEST  TEAM  INTERFACE  * 

I  MANPOWER;  PLANNING;  SAFETY;  MODIFICATION 

INSTALLATION;  INSTRUMENTATION; 

DEMODIFICATION;  CONTRACTOR  INTERFACE 

*  Responses  that  were  added  by  the  interview  respondent. 


Once  the  categories  for  which  negotiations  occurred 
were  identified,  Likert  scale  ratings  (where  one  equalled 
low  negotiability  and  five  equalled  high  negotiability)  were 
solicited  to  determine  how  flexible  in  negotiations  the  Test 
Wing  was  perceived  to  be  by  the  SPO/Lab  managers  and  by  the 


Test  Wing  managers  themselves.  These  ratings  were  also 
averaged  to  compute  mean  values  for  each  category.  Table  5 
presents  the  category  negotiability  rating  mean  values,  the 
percentage  of  managers  responding  in  that  category,  and  the 
range  of  responses. 

TABLE  5 


REPORTING  NEGOTIABILITY  LIKERT  SCALE  MEAN  VALUES, 
PERCENT  RESPONSES,  AND  LIKERT  SCALE  RESPONSE  RANGES 
FOR  THE  TEST  WING  AND  SPO/LAB 


TEST 

WING 

SPO/ 

LAB 

CATEGORY 

MEAN  % 

RES. 

RANGE 

MEAN  % 

RES  . 

RANGE 

PROC/SUPPLY 

3  .  A 

56 

3 

TO 

4 

3  .  3 

33 

2 

TO 

4 

LOGISTICS 

4 . 0 

56 

2 

TO 

5 

3.7 

67 

2 

TO 

5 

FUNDING 

3 . 4 

100 

2 

TO 

4 

3  .  3 

100 

2 

TO 

4 

MANPOWER 

3 . 0 

56 

2 

TO 

5 

3  .  1 

78 

■  2 

TO 

4 

DOCUMENTATION 

3  .  3 

89 

2 

TO 

5 

3  .  3 

44 

2 

TO 

4 

SCHEDULE 

4 . 6 

100 

3 

TO 

5 

4 . 2 

67 

3 

TO 

5 

PLANNING 

4 . 6 

100 

3 

TO 

5 

4 . 2 

67 

3 

TO 

5 

ENGINEERING 

3  .  7 

78 

2 

TO 

5 

3 . 4 

56 

2 

TO 

4 

SAFETY 

3  .  3 

67 

1 

TO 

5 

3  .  7 

67 

2 

TO 

4 

MOD  DESIGN 

3 . 8 

56 

2 

TO 

5 

3 . 2 

56 

2 

TO 

4 

COST 

3  .  1 

100 

2 

TO 

5 

3 . 0 

78 

1 

TO 

5 

MOD  INSTAL'N 

3 . 4 

56 

2 

Xr 

5 

3 . 0 

78 

2 

TO 

4 

A/C  MAINT 

2 . 8 

44 

2 

TO 

4 

3 . 3 

33 

2 

TO 

4 

TEST  ITEM 

3 . 6 

78 

2 

TO 

5 

4 . 0 

67 

3 

TO 

5 

INSTRUMENT 'N 

3 . 9 

78 

3 

TO 

5 

4 . 2 

67 

3 

TO 

5 

FLT  TESTING 

4 . 1 

100 

1 

TO 

5 

4 . 4 

89 

3 

TO 

5 

TECH  PERFORM 

4  .  1 

78 

3 

TO 

5 

4 . 2 

67 

3 

TO 

5 

RANGE  SPT 

3 . 8 

44 

3 

TO 

5 

4 . 0 

56 

2 

TO 

5 

DATA  ANALYSIS 

3 . 6 

89 

2 

TO 

5 

3 . 4 

78 

2 

TO 

5 

DEMOD 

3.0 

56 

2 

TO 

4 

3 . 5 

44 

3 

TO 

4 

CONTR’R  INT 

4 . 6 

56 

4 

TO 

5 

3.9 

78 

3 

TO 

5 

PREFAB 

3  .  7 

33 

2 

TO 

5 

3  .  7 

33 

3 

TO 

4 
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Figure  5.  Reporting  Negotiability  Mean  Comparison 
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Figure  5,  continued.  Reporting  Negotiability  Mean 
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In  analyzing  the  results  in  Table  5  and  Figure  5,  all 
of  the  interview  respondents  (100%  from  the  Test  Wing  and 
100%  from  the  SPOs/Labs)  perceived  that  reporting 
information  in  the  category  of  funding  had  been  negotiated 
between  them.  All  of  the  Test  Wing  managers  felt  that 
reporting  information  in  the  categories  of  schedule, 
planning,  cost,  and  flight  testing  had  been  negotiated 
while  not  all  of  their  SPO/Lab  counterparts  felt  the  same. 
Test  Wing  managers  have  a  higher  mean  value  rating  for  11  of 
the  22  categories.  The  categories  of  documentation  and 
prefab  have  the  same  mean  rating  from  Test  Wing  and  SPO/Lab 
managers.  Having  the  same  mean  rating  suggests  that  the 
counterpart  managers  have  the  same  perception  about  how 
flexible  the  Test  Wing  manager  is  in  negotiating  reporting 
requirements  for  these  two  categories.  Both  Test  Wing  and 
SPO/Lab  managers  have  mean  ratings  of  3.3  for  the  Test 
Wing's  flexibility  in  reporting  information  in  the  category 
of  documentation.  Test  Wing  and  SPO/Lab  managers  have  mean 
ratings  of  3.7  for  the  category  of  prefab.  The  category 
receiving  the  lowest  negotiability  rating  from  SPO/Lab 
managers  is  cost. 

In  analyzing  the  reporting  negotiability  data  via 
statisitical  tests  of  significance  (as  was  done  for  the 
information  importance  area),  normality  assumptions  were 
violated  for  lA  of  22  categories.  Consequently, 
nonparametr i c  rank  sum  tests  were  performed  instead  of  two- 
sample  t  tests.  The  null  hypothesis  for  the  nonpar ame t r i c 


test  is  that  the  two  populations  of  Likert  scale  responses 
for  reporting  negotiability  are  considered  identical.  The 
alternate  hypothesis  is  that  the  two  populations  are  not 
identical.  Statistical  nonparametr i c  analysis  using  the 
rank  sum  test  of  the  STATISTIX  analysis  program  yielded  p 
values  greater  than  0.05  for  all  categories.  Invoking  the 
decision  rule  not  to  reject  the  null  hypothesis  based  on  p 
values  greater  than  0.05  assures  one  with  95%  confidence 
that  the  population  distribution  of  Likert  scale 
negotiability  responses  from  the  Test  Wing  managers  is 
identical  to  the  population  distribution  of  SPO/Lab 
responses.  Having  identical  population  distributions  of 
Likert  scale  negotiability  responses  means  that  interview 
respondents  on  both  sides  of  a  program  have  very  similar 
perceptions  about  the  flexibility  displayed  by  Test  Wing 
managers  in  negotiating  over  SPO/Lab  reporting  requirements. 

Overall  Reporting  Performance 

The  third  area  to  be  discussed  is  that  of  the  overall 
reporting  performance  of  the  Test  Wing  as  perceived  by  its 
SPO/Lab  customers  and  by  the  Test  Wing  test  managers 
themselves.  In  the  same  manner  as  before,  this  area  is 
analyzed  using  descriptive  statistics  (mean  values)  and 
inferential  statistics  (via  nonpar ame t r i c  tests). 

Overall  reporting  performance  data  was  obtained  by 
having  the  test  manager  view  a  carbon  copy  of  the  list 
yielded  from  previous  questions  identifying  categories 
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applicable  to  the  test  program.  As  the  manager  reviewed 
this  list,  he  or  she  was  asked  to  mark  through  those 
categories  in  which  no  reporting  occurred.  When  comparing 
the  two  resulting  lists  for  a  particular  program,  once 
again  there  were  categories  for  which  there  was  no  match 
between  Test  Wing  responses  and  SPO/Lab  responses.  Table  6 
shows  those  categories  by  program  for  which  there  was  no 
match  between  managers'  responses.  None  of  the  categories 
added  by  a  manager  matched  the  responses  of  his  or  her 
counterpart . 


TABLE  6 

MISMATCHES  BETWEEN  COUNTERPART  TEST  MANAGERS 
PERCEPTIONS  ABOUT  REPORTED  INFORMATION 
(OVERALL  REPORTING  PERFORMANCE) 


PROGRAM  MISMATCHED  CATEGORIES 

A  SAFETY;  FLIGHT  TESTING;  DATA  ANALYSIS 

B  PROCUREMENT/SUPPLY;  SUBCONTRACTOR  INTERFACE  *; 

FINAL  REPORTING  * 

C  PROCUREMENT/SUPPLY;  MANPOWER;  ENGINEERING; 

AIRCRAFT  MAINTENANCE;  FLIGHT  PROFILES  * 


D  LOGISTICS;  ENGINEERING;  SAFETY;  MODIFICATION 

DESIGN;  TEST  ITEM;  CONTRACTOR  INTERFACE 

E  PROCUREMENT/SUPPLY;  LOGISTICS;  AIRCRAFT 

MAINTENANCE;  RANGE  SUPPORT;  DEMODIFICATION; 
CHASE  PLANE  *;  TECH  PHOTO  * 

F  DEMODIFICATION  REQUIRED  BY  EXTERNAL  FACTORS  * 

G  PROCUREMENT/SUPPLY;  DEMODIFICATION  REQUIRED 

BY  EXTERNAL  FACTORS  *;  CONTRACTOR  FLIGHT 
TRAINING  * 


[TABLE  CONTINUES ] 
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[TABLE  CONTINUED] 


PROGRAM  MISMATCHED  CATEGORIES 

H  PROCUREMENT/SUPPLY;  DOCUMENTATION;  SCHEDULE; 

PLANNING;  COST;  INSTRUMENTATION;  FLIGHT 
TESTING;  TECHNICAL  PERFORMANCE;  DATA 
ANALYSIS;  CONTRACTOR  INTERFACE;  COMBINED  TEST 
TEAM  INTERFACE  * 

I  PROCUREMENT/SUPPLY;  LOGISTICS;  DOCUMENTATION; 

PLANNING;  ENGINEERING;  SAFETY;  MODIFICATION 
DESIGN;  TEST  ITEM;  FLIGHT  TESTING;  TECHNICAL 
PERFORMANCE;  DATA  ANALYSIS;  DEMODIFICATION; 
CONTRACTOR  INTERFACE;  FAA  INTERFACE 
(SIMULATOR  CERTIFICATION)  * 

*  Responses  that  were  added  by  the  interview  respondent. 


After  the  categories  in  which  reporting  had  occurred 
were  identified,  the  SPO/Lab  managers  were  asked  to  give 
their  perceptions  of  how  well  the  Test  Wing  performed 
overall  in  reporting  information  for  each  category.  Also, 
Test  Wing  managers  were  asked  for  their  perceptions  of  how 
well  they  felt  their  own  overall  reporting  performance  was. 
These  perceptions  were  given  by  the  interview  respondents  in 
a  Likert  scale  rating  where  one  equalled  poor  and  five 
equalled  outstanding.  Mean  values  were  computed  from  the 
responses  for  each  category.  Table  7  shows  the  mean  values, 
percentages  of  responses,  and  ranges  for  each  category. 

As  shown  in  Table  7,  the  only  category  in  which  all  of 
the  managers  from  the  Test  Wing  and  the  SPOs/Labs  assigned 
an  overall  performance  rating  is  funding.  No  other 
categories  received  a  100%  response  from  the  SPOs/Labs.  All 
of  the  Test  Wing  managers  interviewed  assigned  perception 
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TABLE  7 


OVERALL  REPORTING  PERFORMANCE  LIKERT  SCALE  MEAN  VALUES, 
PERCENT  RESPONSES,  AND  LIKERT  SCALE  RESPONSE  RANGES 
FOR  THE  TEST  WING  AND  SPO/LAB 


TEST  WING  SPO/LAB 


CATEGORY 

MEAN 

%  RES  . 

RANGE 

MEAN 

%  RES  . 

RANGE 

PROC/SUPFLY 

2 . 8 

67 

2 

TO 

A 

3  .  7 

78 

2 

TO 

5 

LOGISTICS 

3  .  7 

78 

2 

TO 

5 

A  .  0 

67 

3 

TO 

5 

FUNDING 

3  .  7 

100 

1 

TO 

5 

A  .  2 

100 

3 

TO 

5 

MANPOWER 

3 . 9 

78 

2 

TO 

5 

3 . 6 

89 

3 

TO 

5 

DOCUMENTATION 

3 . 6 

89 

2 

TO 

5 

A  .  3 

67 

A 

TO 

5 

SCHEDULE 

A  .  6 

100 

3 

TO 

5 

A  .  A 

89 

2 

TO 

5 

PLANNING 

A  .  7 

100 

A 

TO 

5 

A  .  A 

78 

2 

TO 

5 

ENGINEERING 

3  .  7 

100 

3 

TO 

5 

A  .  2 

67 

3 

TO 

5 

SAFETY 

A  .  1 

100 

3 

TO 

5 

A  .  5 

67 

3 

TO 

5 

MOD  DESIGN 

3 . 6 

89 

2 

TO 

5 

A  .  2 

67 

3 

TO 

5 

COST 

3  .  7 

100 

1 

TO 

5 

3 . 8 

89 

1 

TO 

5 

MOD  INSTAL'N 

3.9 

89 

3 

TO 

5 

3 . 6 

89 

1 

TO 

5 

A/C  MAINT 

3 . 8 

67 

3 

TO 

5 

3.5 

67 

1 

TO 

5 

TEST  ITEM 

A  .  1 

89 

3 

TO 

5 

A  .  0 

67 

3 

TO 

5 

INSTRUMENT 'N 

A  .  0 

89 

2 

TO 

5 

3 . 9 

78 

3 

TO 

:) 

FLT  TESTING 

A  .  8 

100 

A 

TO 

5 

A  .  5 

67 

2 

TO 

5 

TECH  PERFORM 

A  .  6 

89 

3 

TO 

5 

A  .  3 

67 

A 

TO 

5 

RANGE  SPT 

A  .  3 

AA 

A 

TO 

5 

A  .  6 

56 

A 

TO 

5 

DATA  ANALYSIS 

A  .  6 

100 

3 

TO 

5 

A  .  2 

67 

3 

TO 

5 

DEMOD 

3.7 

67 

3 

TO 

5 

3  .  7 

67 

1 

TO 

5 

CONTR'R  INT 

A  .  A 

100 

3 

TO 

5 

A  .  2 

67 

3 

TO 

5 

PREFAB 

3 . 5 

67 

2 

TO 

5 

3  .  A 

56 

2 

TO 

A 

ratings  about  their  performance  in  the  categories  of 
schedule,  planning,  engineering,  safety,  cost,  flight 
testing,  data  analysis,  and  contractor  interface  (and 
funding  as  mentioned  above).  Mean  values  for  Test  Wing 
responses  were  higher  than  SPO/Lab  responses  for  12  of  the 
categories  and  lower  for  nine  of  the  responses.  There  was 
one  tie  between  mean  ratings  at  3.7  for  the  category  of 
demod i f i ca t i on .  Mean  ratings  varied  from  one  to  five  within 
each  management  group.  The  largest  difference  between  mean 


values  was  0.9  for  the  category  of  procurement/supply. 
Eighteen  of  the  22  categories  are  within  a  0.5  difference  in 
mean  values;  these  categories  are  logistics,  funding, 
manpower,  schedule,  planning,  engineering,  safety,  cost, 
modification  installation,  aircraft  maintenance,  test 
i tern , i ns t r umenta t i on ,  flight  testing,  technical  performance, 
range  support,  demodification,  contractor  interface,  and 
prefab . 

Figure  6  provides  mean  value  comparisons  in  graphic 
form  of  the  Likert  scale  responses  of  Test  Wing  overall 
reporting  performance  perceptions. 

Statistical  tests  of  significance  were  performed  on  the 
overall  reporting  performance  Likert  scale  response  data  (as 
was  done  for  the  areas  of  information  importance  and 
reporting  negotiability).  In  this  case,  the  normality 
assumption  was  violated  in  13  of  22  categories.  For  this 
reason,  the  rank  sum  nonparametr i c  test  was  applied  instead 
of  the  parametric  two-sample  t  test.  In  all  categories,  the 
p  values  were  over  the  0.05  threshold,  signifying  that  the 
null  hypothesis  was  not  to  be  rejected.  The  null  hypothesis 
is  that  the  population  distributions  of  overall  reporting 
performance  Likert  scale  responses  from  the  Test  Wing  and 
the  SPOs/Labs  are  considered  identical.  Not  rejecting  the 
null  hypothesis  indicates  that  the  perceptions  held  by  Test 
Wing  managers  and  SPO/Lab  managers  about  the  overall 
reporting  performance  of  the  Test  Wing  are  very  similar. 
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Figure  6.  Overall  Reporting  Performance  Mean  Comparison 


Figure  4,  continued. 
Compar i son 


Methods  of  Communication 

Each  interview  respondent  was  asked  what  he  or  she 
perceived  to  be  the  most  commonly  used  methods  of 
communication  for  each  of  the  categories  of  information  in 
which  reporting  had  occurred.  The  following  six  methods 
were  identified  by  the  managers:  telephone,  written,  group 
meetings,  dialog  (i.e.,  face-to-face  conversation  between 
two  people),  fax,  and  computer  link.  Similarities  and 
differences  exist  between  what  communication  methods  the 
Test  Wing  and  SPO/Lab  counterpart  managers. perceived  as  most 
commonly  used  by  the  Test  Wing  in  reporting  information. 

The  managers'  responses  are  presented  in  Table  8  by  category 
and  by  program.  Since  reporting  information  was  often  done 
using  more  than  one  method,  the  primary  response  (i.e.,  the 
method  considered  most  commonly  used)  is  listed  first  in 
Table  8,  and  other  responses  follow. 

TABLE  8 

INTERVIEW  RESPONSES  OF  THE  MOST  COMMONLY  USED 
METHODS  OF  COMMUNICATING  TEST  INFORMATION 

NOTE--The  legend  is  as  follows:  T  is  telephone,  W  is 
written,  M  is  group  meetings,  D  is  dialog,  F  is  fax,  C  is 
computer  link-up,  and  an  asterisk  (*)  means  managers 
responded  that  there  was  no  reporting  done  in  that  category. 


CATEGORY 

TW 

SPO/LAB 

CATEGORY 

TW 

SPO/LAB 

PROCUREMENT/ 

''SUPPLY 

LOGISTICS 

Program  A: 

T 

T 

Program  A: 

T/F 

T 

B  : 

★ 

T 

B  : 

T/M 

T 

C  ; 

T 

* 

C  ; 

T/W 

T 

D  : 

T/M 

W 

D; 

T/M 

★ 
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CATEGORY 

TW 

SPO/LAB 

PROCUREMENT/SUPPLY 

Program 

E 

* 

T 

F 

T 

T 

G 

T/D 

T 

H 

★ 

W/T 

I 

T/W 

★ 

FUNDING 

Program 

A 

W 

M/D/W 

B 

M/W 

T/W 

C 

W 

W 

D 

T/M/W 

W/T 

E 

W/T 

W 

F 

T/W 

W 

G 

T/D 

T/W 

H 

W 

W/T 

I 

W 

W/T 

DOCUMENTATION 

Program 

A 

T/W 

D/M 

B 

W/M/T 

W 

C 

W/M 

W 

D 

★ 

★ 

E 

W 

W 

F 

T/W 

w 

G 

T/W 

T/W 

H 

W 

★ 

I 

T/W 

*■ 

PLANNING 

Program 

A 

M 

W/T/D 

B 

W 

W 

C 

D 

M/W 

D 

M/T/F 

T/D 

E 

W/T/D 

W 

F 

T/M 

D 

G 

T 

D/M 

H 

M/W 

★ 

I 

W/T 

★ 

SAFETY 

Program 

A 

F/T/W 

★ 

B 

T 

W 

C 

T 

F/W 

D 

M/T 

★ 

E 

W/T/M 

T/W 

F 

T/W 

D 

CATEGORY 

TW 

SPO/LA 

LOGISTICS 

Program  E 

★ 

T 

F 

T 

T 

G 

T/D 

T 

H 

★ 

W/T 

I 

T/D 

★ 

MANPOWER 

Program  A 

T 

D/T 

B 

D 

M 

C 

★ 

T 

D 

★ 

★ 

E 

W/T 

W/T 

F 

T 

W/T 

G 

T/D 

T 

H 

T/W 

W/T 

I 

W 

W 

SCHEDULE 

Program  A 

T 

W/T/D 

B 

M/W 

M/T 

C 

D/M 

M/W 

D 

M/T/F 

T/D 

E 

W/T 

T/W 

F 

T/W 

D 

G 

T/W 

D/W/T 

H 

W 

I 

T/W 

W/T 

ENGINEERING 

Program  A 

T/F 

W/T/D 

B 

M/W 

M 

C 

T 

★ 

D 

M/T 

★ 

E 

W/M 

T/W 

F 

T/M 

D 

G 

T 

D 

H 

T/W 

W/T 

I 

W 

★ 

MODIFICATION  DESIGN 

Program  A 

T 

M 

B 

M 

M 

C 

M 

T 

D 

M/T 

★ 

E 

D 

T 

F 

M 

D/M 
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CATEGORY 

TW 

SPO/LAB 

CATEGORY 

TW 

SPO/LA 

SAFETY 

MODIFICATION  DESIGN 

Program 

G 

T/W 

T 

Program  G 

T 

W 

H 

W 

W/T 

H 

★ 

★ 

I 

W/T 

★ 

I 

W 

★ 

COST 

MODIFICATION  INSTALLATION 

Program 

A; 

T/W 

D/W 

Program  A 

T/D 

D 

B; 

M/W 

W 

B 

M 

M 

C ; 

W 

W 

C 

M 

T/D 

D: 

M/T/W 

W/T 

D 

M/T 

T/D 

E: 

W 

W 

E 

D 

T 

F; 

T/W/F 

T/W 

F 

T 

D 

G: 

T/W 

W/T 

G 

T 

T/M 

H: 

W/T 

*• 

H 

* 

★ 

I 

T/W 

W 

I 

T/D 

T 

AIRCRAFT  MAINTENANCE 

TEST  ITEM 

Program 

A 

T 

T 

Program  A 

T/W 

M/D/T 

B 

T/M 

D 

B 

M/T/W 

M 

C 

W 

★ 

C 

M 

T/D 

D 

M/T/F 

T/D 

D 

M/T/F 

* 

E 

*• 

T/W 

E 

D/W 

T/W 

F 

T 

W/T 

F 

T 

T/D 

G 

T 

W/T 

G 

T 

T 

H 

* 

★ 

H 

★ 

* 

I 

* 

I 

T/D 

★ 

INSTRUMENTATION 

FLIGHT  TESTING 

Program 

A 

T/M/D 

M/T 

Program  A 

T/W 

★ 

B 

M 

M 

B 

D/M/W 

T/M 

C 

* 

*■ 

C 

W 

C/W 

D 

M/T 

T/D 

D 

M/T/F 

T/D 

E 

W 

•T 

E 

W/D 

T/W 

F 

T/W 

D/T 

F 

T 

T 

G 

T 

D/M 

G 

T 

T/W 

H 

T 

★ 

H 

M/T 

★ 

I 

W/D 

W/M 

I 

T/W 

★ 

TECHNICAL 

PERFORMANCE 

RANGE  SUPPORT 

Program 

A 

T/F 

T 

Program  A 

★ 

★ 

B 

D/M/W 

M 

B 

M/W 

M 

C 

W 

T/D/W 

C 

★ 

★ 

D 

★ 

★ 

D 

M/T/F 

★ 

E 

W 

T/W 

E 

♦ 

T 

F 

M 

D/T 

F 

T/M 

D/T 

G 

T/W 

T/W 

G 

T 

D/M/T 

H 

W 

★ 

H 

★ 

W/T 

I 

T/W 

★ 

I 

★ 

★ 
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CATEGORY  TW  SPO/LAB  CATEGORY  TW  S  PO/LAB 


DATA  ANALYSIS 

DEMODIFICATION 

Program  A 

M/W 

★ 

Program 

A 

T/W 

M/D/W 

B 

M 

W 

B 

T 

D 

C 

W 

W/T 

C 

* 

★ 

D 

M/T/W 

T 

D 

M/T 

T/D 

E 

W 

T/W 

E 

*• 

T 

F 

T 

D  /M 

F 

T/W 

T 

G 

W/D 

W 

G 

T 

W 

H 

W 

★ 

H 

★ 

* 

I 

T/W 

* 

I 

W/T 

* 

CONTRACTOR  INTERFACE 

PREFAB 

Program  A 

M/T/F/W 

D/T 

Program 

A 

T 

D/T 

B 

M/T/D/W 

M/T 

B 

T/M 

M 

C 

T/W/M/F 

T/D 

C 

★ 

★ 

D 

M/T 

* 

D 

M/T/F 

T/D 

E 

T/D 

T 

E 

* 

★ 

F 

T 

T/M 

F 

T/W 

D/M 

G 

T 

T/D 

G 

T 

T/W 

H 

T/W 

★ 

H 

* 

★ 

I 

D/W 

★ 

I 

T 

★ 

The  responses  shown  in  Table  8  will  be  analyzed  from 
two  per spect i ves--by  category  and  by  program.  Discussion 
will  first  focus  on  comparison  of  responses  within 
categories,  and  then  comparisons  will  be  made  between 
managers  for  specific  programs. 

In  comparing  only  the  primary  communication  method 
named  by  the  managers  (i.e.,  those  methods  identified  as 
most  commonly  used  by  the  Test  Wing  in  reporting 
information),  at  least  one  match  between  responses  of  Test 
Wing  managers  and  SPO/Lab  managers  occurred  for  each 
category.  At  most  there  are  five  method  matches  between 
responses  from  counterpart  managers  for  a  particular 
category.  Categories  for  which  five  method  matches  occurred 


57 


are  logistics,  funding,  and  contractor  interface. 

Categories  in  which  the  least  number  of  method  matches 
occurred  for  a  program  were  schedule,  engineering, 
modification  design,  aircraft  maintenance,  range  support, 
demodification,  and  prefab.  For  each  of  these  categories 
only  one  program  had  corresponding  responses  between 
managers . 

In  comparing  responses  by  program,  Test  Wing  managers 
and  SPO/Lab  managers  of  Program  G  specified  the  same  primary 
method  of  communication  for  13  of  the  22  categories.  This 
was  the  highest  number  of  matches  for  any  program.  The  next 
highest  number  of  method  matches  was  11  for  Program  B, 
followed  by  seven  for  Program  F.  Programs  C  and  E  both  had 
six  method  matches,  and  Programs  A  and  I  also  tied  with  four 
matches  each.  Program  H  had  only  two  method  matches,  and 
Program  D  had  no  method  matches  between  Test  Wing  managers 
and  SPO/Lab  managers.  A  match  between  responses  indicates 
that  counterpart  managers  perceived  the  same  communication 
method  as  the  one  moat  commonly  used  in  reporting 
i nf orma  t i on . 

Communication  by  the  methods  of  telephone,  group 
meetings,  fax,  and  computer  link  messages  from  another 
terminal  were  use  I  in  reporting  information  as  the  name  of 
each  method  implies.  The  dialog  method  was  described  during 
the  interviews  as  face-to-face  conversation  between  the  Test 
Wing  manager  and  the  counterpart  SPO/Lab  manager  in  which 
reporting  occurred.  The  written  method  took  manv  forms 
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depending  on  the  category  for  which  it  was  used.  Table  9 
lists  the  specific  forms  in  which  the  written  method  of 
communication  was  manifest. 


TABLE  9 

SPECIFIC  FORMS  OF  WRITTEN  COMMUNICATION 
USED  IN  REPORTING  INFORMATION 


PROGRAM/ 

CATEGORY  TEST  WING  SPO/LAB 


Program  A: 

FUNDING 

DOCUMENTATION 

SAFETY 

COST 

TEST  ITEM 
FLIGHT  TESTING 
DATA  ANALYSIS 
DEMODIFICATION 
CONTRACTOR  INT 


Statement  of  Capability  (SC)  Report 

Sketches  - 

Letters  - 

Formal  Report  Report 

Final  Report  - 

Formal  Report  - 

Formal  Report  - 

SC  - 

Reports  --- 


Program  B: 

FUNDING 

update 

DOCUMENTATION 

PLANNING 

COST 

FLIGHT  TESTING 
RANGE  SUPPORT 
DATA  ANALYSIS 
CONTRACTOR  INT 


Informal  Data  Doc 
Letters 

Letters 


Prgm  Intro  Doc 

Action  Items 
Test  Plan  Doc 
Informal  Cost  Doc 


Formal  Report 


Program  C : 

LOGISTICS 

FUNDING 

PLANNING 

COST 

FLIGHT  TESTING 

Program  D; 
PROCUREMENT /SUP PLY 
FUNDING 
COST 

DATA  ANALYSIS 


Letter  follow-up 
SC 


Quick  Look  Reports 


JOCA5  Mthly  Stmt 
JOCAS  Mthly  Stmt 
Raw  Data 


Meet ing  Notes 
JOCAS  Reports 
Qu ick  Look  Rots 


Month Iv  Cost  Rpt 
Monthly  Cost  Rpt 
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PROGRAM/ 

CATEGORY 

Program  E: 

COST 

Program  F: 

FUNDING 

DOCUMENTATION 

SCHEDULE 

COST 

INSTRUMENTATION 

DEMODIFICATION 

PREFAB 

Program  G: 

FUNDING 
DOCUMENTATION 
SCHEDULE 
MOD  DESIGN 
COST 

A/C  MAINTENANCE 
FLIGHT  TESTING 
TECH  PERFORMANCE 
DATA  ANALYSIS 
DEMODIFICATION 
PREFAB 

Program  H: 

PROCUREMENT/SUPPLY 

FUNDING 

MANPOWER 

ENGINEERING 

SAFETY 

COST 

RANGE  SUPPORT 
DATA  ANALYSIS 
CONTRACTOR  INT 

Program  I : 

PROCUREMENT/SUPPLY 

FUNDING 

MANPOWER 

DOCUMENTATION 

SCHEDULE 

PLANNING 

ENGINEERING 

SAFETY 

MOD  DESIGN 

COST 


TEST  WING 


SPO/LAB 

JOCAS 


Letter 

■ 

Letter 

— 

Letter 

— 

Letter 

Monthly  Reports 

Letter 

— 

Letter 

— 

Let  ter 

— 

Records 

— 

JOCAS 

JOCAS 

_ 

SC 

_ 

Changes 

^  ^  ^ 

Drawings 

^  — 

Sched  (Routine) 

_  —  — 

Prog/Final  Rpts 

—  —  — 

Quick  Look  Rpts 

_ 

Quick  Look  Rpts 

_ 

S  t a  t emen t  of  Cap 

_ 

Drawings 

Mthly 

Status 

Rpts 

Report 

Mthly 

Status 

Rpts 

SC 

Mthly 

Status 

Rpts 

_ _ 

Mthly 

Status 

Rpts 

JOCAS 

Mthly 

Status 

Rpts 

Letter 

Letter 

Mt  h  1 V 

Status 

Rpts 

Test  Concept  Doc 

(TCD)  - 

TCD/SC 

Repor  t /Le  1 1  e  r 

TCD/SC 

JOCAS  Report 

Prog/Final  Report 

— 

Progress  Reports 

Planning  Docs 

TCD/Test  Plan 

— 

Test  PI  an 

-  —  - 

Test  Plan 

— 

Test  Plan 

— 

JOCAS/SC 

JOCAS 

[ TABLE 
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PROGRAM/ 

CATEGORY  TEST  WING  SPO/LAB 

Program  I  (continued): 

INSTRUMENTATION  TCD/Prog  Rpt/Final  Rpt  SC/Letters 

FLIGHT  TESTING  Prog  Report/Final  Rpt  - 

TECH  PERFORMANCE  Prog  Report/Final  Rpt  - 

DATA  ANALYSIS  Prog  Report/Final  Rpt  - 

DEMODIFICATION  Prog  Report/Final  Rpt  - 

CONTRACTOR  INT  Prog  Report/Final  Rpt  - 


There  were  only  a  few  cases  in  Table  9  where 
counterpart  managers  on  a  program  specified  similar  written 
responses.  While  some  of  the  written  responses  use 
different  vocabulary,  they  denote  the  same  method  of 
communication.  Program  A  managers  responded  similarly  for 
the  communication  method  used  in  reporting  cost  information. 
The  Test  Wing  manager  said  "Formal  Reports"  were  the  most 
commonly  used  method,  while  the  SPO/Lab  manager  called  it 
"Report."  Program  B  had  no  matching  written  responses. 

Both  managers  for  Program  C  said  "Quick;  Look  Reports"  were 
the  most  commonly  used  methods  in  reporting  flight  test 
information.  The  Test  Wing  manager  for  Program  D  said  that 
the  "JOCAS  Monthly  Statement”  (i.e.,  Job  Order  Cost 
Accounting  System  [JOCAS])  was  most  commonly  used  to  report 
cost  i  nf  orma  t  i ->n ,  and  the  Program  D  SPO/Lab  manager  said 
"Monthly  Cost  Reports"  were  used.  Program  E  had  no  matches 
between  respondents  for  written  methods  of  common i ca t i on . 
Program  F  also  had  no  written  method  matches.  The 
responses  from  managers  on  Program  G  coincided  for  the 
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category  of  documentation,  where  both  managers  said  "JOCAS" 
was  the  most  commonly  used  method  of  reporting.  For  Program 
H,  the  responses  for  the  category  of  funding  closely  matched 
as  the  Test  Wing  manager  called  his  method  of  reporting 
"Written  Report",  and  the  SPO/Lab  manager  recognized  it  as 
"Monthly  Status  Reports."  There  were  two  matches  in  Program 
I  for  written  commun i ca t ion  methods.  Both  the  Test  Wing 
manager  and  the  SPO/Lab  manager  said  cost  was  reported  using 
"JOCAS."  For  the  category  of  instrumentation,  the  Program  I 
managers'  responses  were  similar  in  that  the  Test  Wing 
manager  used  the  "Test  Concept  Document"  to  report 
information,  and  the  SPO/Lab  manager  said  that  information 
was  reported  uring  the  "Statement  of  Capability"  which  the 
Test  Wing  manager  said  was  part  of  the  Test  Concept 
Document . 

Further  comparisons  can  be  made  between  the  responses 
of  the  managers  by  examining  the  frequencies  at  which  the 
methods  were  specified.  Table  10  below  shows  four  of  the 
methods  that  Test  Wing  managers  used  to  report  information 
to  their  counterpart  managers  at  the  SPOs  and  Labs  and  the 
frequency  of  responses  for  each  category.  The  two  other 
communication  methods  that  were  identified  by  the  interview 
respondents  were  fax  and  computer  link.  Fax  was  only 
mentioned  twice  (for  the  category  of  safety),  and  computer 
link  was  only  mentioned  once  (for  the  category  of  flight 
testing),  so  they  are  not  included  in  Table  10. 


62 


TABLE  10 


FREQUENCY  COUNTS  FOR  METHODS  OF  COMMUNICATION 


METHODS 

AND 

FREQUENCY 

FOR 

EACH  METHOD 

TELEPHONE 

WRITTEN 

MEETINGS 

DIALOG 

CATEGORY 

TW 

SPO 

TW 

SPO 

TW 

SPO 

iw 

SP( 

PROC/SUPPLY 

6 

5 

- 

2 

- 

- 

- 

~ 

LOGISTICS 

7 

6 

- 

1 

- 

- 

- 

- 

FUNDING 

3 

2 

5 

6 

1 

1 

- 

- 

MANPOWER 

4 

2 

2 

4 

- 

1 

1 

1 

DOCUMENT ■ N 

4 

1 

4 

4 

- 

- 

- 

1 

SCHEDULE 

4 

2 

2 

2 

2 

2 

1 

2 

PLANNING 

2 

1 

3 

3 

3 

1 

1 

2 

ENGINEERING 

5 

1 

2 

1 

2 

1 

- 

2 

SAFETY 

4 

2 

2 

2 

1 

- 

- 

1 

MOD  DESIGN 

2 

2 

1 

1 

4 

2 

1 

1 

COST 

4 

1 

3 

6 

2 

- 

- 

1 

MOD  INSTAL’N 

4 

5 

- 

- 

3 

1 

1 

2 

A/C  MAINT 

4 

3 

1 

2 

1 

- 

- 

1 

TEST  ITEM 

4 

4 

- 

- 

3 

2 

1 

- 

INSTRUMENT ' N 

4 

2 

2 

1 

2 

2 

- 

2 

FLT  rE.brrNG 

4 

5 

2 

- 

2 

- 

1 

- 

TECH  PERFORM 

3 

4 

3 

- 

1 

1 

1 

1 

RANGE  SPT 

2 

1 

- 

1 

2 

1 

- 

2 

DATA  ANALYSIS 

2 

2 

4 

3 

3 

- 

- 

1 

DEMOD 

4 

3 

1 

1 

1 

1 

- 

1 

CONTR'R  INT 

5 

4 

- 

- 

3 

1 

1 

1 

PREFAB 

5 

2 

_ 

_ 

1 

i 

_ 

2 

TOTAL 

86 

60 

37 

40 

37 

1  8 

9 

24 

Table  10  totals  show  that  telephone  communication  was 
spee  fied  more  times  by  the  Test  Wing  (86)  or  the  SPOs/Labs 
(60)  than  any  other  method.  The  written  method  of 
communication  was  the  next  method  mentioned  most  often  bv 
the  SPOs  and  Labs  (40);  however,  there  was  a  tie  between  the 
number  of  times  Test  Wing  managers  specified  the  written 
method  (37)  and  the  meetings  method  (37).  SPOs/Labs 
identified  the  dialog  method  (24)  more  often  than  the 
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meeting  method,  and  the  Test  Wing  specified  the  meeting 
method  (37)  more  than  the  dialog  method  (9). 


Reporting  Issues 

Each  interview  respondent  was  asked  to  share  any 
comments  about  the  reporting  aspects  of  their  test  program. 
Issues  raised  by  managers  in  their  open-ended  responses  are 
discussed  by  program  in  the  following: 

Program  A.  Managers  from  both  sides  of  this  program 
expressed  that  they  had  a  better  working  relationship  than 
others  due  to  the  classified  nature  of  the  program  and  the 
uniqueness  of  the  testing  unit  within  the  Test  Wing 
organization.  With  regard  to  reporting  information,  the 
SPO/Lab  manager  made  an  assumption  that  the  Test  Wing  was 
going  to  determine  what  his  information  needs  were  (i.e.,  he 
did  not  think  it  was  necessary  to  make  all  his  information 
needs  known  to  the  Test  Wing). 

Program  B.  The  SPO/Lab  manager  for  this  program  made 
specific  comments  about  the  formal  cost  reports  he  received 
from  the  Test  Wing.  He  described  these  reports  as  "poor" 
and  "worthless";  however,  he  assigned  the  highest  score  (5) 
in  rating  the  Test  Wing  in  the  overall  reporting  performance 
area.  He  explained  that  even  though  the  formal  cost  reports 
were  poor,  his  counterpart  manager  in  the  Test  Wing  provided 
"current"  estimates  of  cost  "orfline"  to  meet  his 
information  needs. 
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Program  C.  The  Test  Wing  manager  described  this 
program  as  being  different  in  that  it  was  managed  by  a 
contractor  instead  of  the  Test  Wing.  The  Test  Wing  does, 
however,  have  several  Test  Wing  people  dedicated  to  the 
program  and  reports  information  to  its  SPO/Lab  customer.  In 
reporting  information  to  the  SPO/Lab,  the  Test  Wing  manager 
said  that  he  talks  to  different  people  at  the  SPO/Lab  on 
different  test  i ssues--meaning  that  the  SPO/Lab  manager  is 
not  his  single  point  of  contact,  although  the  Test  Wing 
manager  keeps  his  customer  informed.  The  SPO/Lab  manager, 
on  the  other  hand,  perceived  himself  as  the  focal  point 
recipient  of  Test  Wing  reporting. 

The  SPO/Lab  manager  also  said  that  the  program  was 
different  from  others  in  that  the  Test  Wing  does  not  funtion 
as  a  classical  Responsible  Test  Organization  (RTO).  The 
nature  of  the  program  is  more  commercial  than  military.  The 
SPO/Lab  manager  commented  about  the  Job  Order  Cost 
Accounting  System  (JOCAS)  as  lagging  too  far  behind  funding 
and  difficult  to  interpret.  He  recommended  supplementing 
the  cost  report  with  useful  information.  This  was  the  only 
program  in  which  some  information  was  said  to  have  been 
reported  using  computer  link.  The  SPO/Lab  manager  described 
the  use  of  computer  link  to  send  initial  flight  test 
information  electronically  from  the  contractor  test  site  in 
the  form  of  Quick  Look  reports. 

Program  D.  The  Test  Wing  manager  commented  about  the 
monthly  funding  and  cost  status  reports,  saving  that  thev 


65 


were  submitted  to  his  customer  from  another  unit  within  the 
Test  Wing  ( RM )  and  not  from  himself. 

The  SPO/Lab  manager  commented  that  he  does  receive  cost 
information  on  a  monthly  basis  (not  JOCAS )  and  sometimes  by 
phone  from  the  Test  Wing,  but  that  there  is  no  wav  to 
identify  the  cost  estimates.  He  said  there  is  no  way  to 
track  expenditures.  He  recommended  that  the  Test  Wing 
implement  using  Cost  Schedule  Control  Systems  Criteria 
(C/SCSC).  Having  managed  the  program  through  many 
generations  of  testing,  the  SPO/Lab  manager  commented  that 
over  the  years  test  management  has  become  more  bureaucratic 
and  less  effective.  He  also  said  that  the  Test  Wing  served 
as  more  of  a  participating  test  organizati-'n  (PTO)  than  a 
responsible  test  organization  (RTO)  on  this  program,  but 
that  the  Test  Wing  still  reported  information  to  him. 

Program  E.  As  with  a  few  of  the  test  programs 
previously  mentioned,  this  program  is  one  in  which  the 
contractor  rather  than  the  Test  Wing  conducted  the  test 
program.  The  Test  Wing  did,  however,  ensure  that  tests  were 
properly  conducted  to  safeguard  Air  Force  interests.  The 
Test  Wing  also  ensured  that  data  collection  by  the 
contractors  was  valid  and  that  contractor  reports  to  the 
SPO/Lab  were  valid.  Thus,  the  Test  Wing  was  responsible  to 
its  customer,  the  SPO/Lab,  to  report  information  about  test 
program  progress. 

Comments  from  the  SPO/Lab  manager  were  about  his 
receipt  of  cost  information.  He  said  that  at  times  the  Test 
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Wing  manager  had  no  information  about  remaining  funds  on  the 
test  program. 

Program  F.  The  Test  Wing  manager  offered  no  comments; 
however,  the  SPO/Lab  manager  commented  on  cost  information. 
The  SPO/Lab  manager  said  that  his  counterpart  manager  in  the 
Test  Wing  was  more  flexible  in  negotiating  to  meet  his 
information  needs  than  were  the  other  people  within  the  Test 
Wing  wiio  also  handled  reporting  cost  information.  The 
SPO/Lab  manager  said  he  thought  in  general  that 
negotiability  depends  on  individual  personality.  He  also 
said  that  he  relied  more  on  cost  information  transmitted  bv 
phone  than  sent  to  him  in  monthly  reports--ma inly  because 
the  reports  were  difficult  to  understand,  and  there  was  no 
bottom  line  costs. 

Program  G.  The  Test  Wing  manager  for  this  program  said 
that  his  counterpart  manager  preferred  to  be  given  program 
status  updates  over  the  telephone  or  in  face-to-face  dialog 
rather  than  in  periodic  reports.  Since  much  of  the 
reporting  was  done  verbally,-  the  Test  Wing  manager  said  that 
he  also  kept  written  records  as  backups  to  what  information 
was  reported.  He  said  this  system  worked  well  for  him.  He 
also  said  that  the  Program  Introduction  document 
requirements  (i.e.,  testing  and  reporting  requirements)  from 
the  SPO/Lab  were  not  as  stringent  for  this  program  as  with 
other  programs.  This  mav  be  a  reflection  of  the  somewhat 
laissez  faire  management  style  of  his  counterpart  manage^' 
in  the  SPO/Lab.  The  Program  G  Test  Wing  manager  also  had 


comments  about  cost  reporting.  He  considered  the  JOCAS 
information  as  not  being  useful  to  his  customer,  so  he  also 
reported  status  on  remaining  funds  and  the  Test  Wing's  rate 
of  spending.  He  also  said  that  his  counterpart  manager  in 
the  SPO/Lab  was  his  focal  point  to  which  he  reported 
information. 

The  SPO/Lab  manager's  comments  corresponded  with  those 
of  the  Test  Wing  manager  about  cost  reporting.  The  SPO/Lab 
manager  said  he  does  not  care  to  search  through  JOCAS 
reports,  so  he  would  rather  call  the  Test  Wing  manager 
directly  for  information  about  cost.  When  rating  the  Test 
Wing  for  negotiability  in  cost  reporting,  the  SPO/Lab 
manager  gave  a  high  rating  of  five  for  his  counterpart 
manager  and  assigned  a  separate  lower  rating  of  three  to  th 
Test  Wing  unit  who  supplies  JOCAS  reports.  The  SPO/Lab 
manager  said  that  a  unique  point  about  his  organisation  is 
that  he  serves  as  tiie  program's  engineer  as  well  as  tne 
manager,  so  there  is  little  confusion  about  who  the  focal 
po i nt  i s . 

Program  H.  The  Test  Wing  manager  said  that  for  this 
program  testing  is  done  by  a  contractor,  but  the  Test  Wing 
concurs  on  flig’nt  test  reports  and  serves  as  the  SPO/Lan's 
monitor  of  contractor  testing  activities. 

The  SPO/Lab  manager  concurred  that  this  was  a 
contractor  conducted  test  program,  and  that  the  Test  Wing 
also  participated.  He  commented  about  the  monthly  status 


reports  he  received  from  the  Test  Wing  by  saying  that  the 
reports  do  not  give  enough  detail  for  him  to  gain  insight. 

Program  I.  The  Test  Wing  manager  identified  his 
primary  counterpart  manager  to  whom  he  reported  information 
as  the  SPO/Lab  .program  manager  instead  of  the  SPO/Lab  test 
manager.  His  feeling  was  that  all  information  should  go  to 
this  customer  focal  point,  but  that  matrix  management 
sometimes  presents  problems  in  distorting  information  that 
eventually  may  get  to  the  program  manager  (focal  point)  if 
he  is  bypassed.  The  Test  Wing  manager  used  a  unique  tool  to 
sketch  out  a  test  program  early  in  its  development  and  to 
assist  the  SPO/Lab  in  their  composition  of  the  Program 
Introduction  document .  This  unique  tool  is  called  the  Test 
Concept  Document  (TCD).  The  Test  Concept  Document  is  early 
documentation  of  the  overall  approach  to  the  test  program, 
and  according  to  the  Test  Wing  manager  of  Program  I,  it 
became  an  attachment  to  the  Statement  of  Capability  provided 
by  the  Test  Wing  to  the  SPO/Lab. 

In  discussing  the  reporting  of  cost  information,  the 
Test  Wing  manager  said  that  managers  do  not  like  the  J0CA5 
system  because  it  is  commonly  three  to  four  months  late, 
difficult  to  intrepret  actual  costs,  and  frequently  has  dat.s 
i nput  errors  . 

The  Test  Wing  manager  said  this  program  could  nave  used 
acces  o  a  local  area  network  (LAN)  for  reporting 
information  via  computer  terminals.  He  specifically 
mentioned  that  Aeronautical  Systems  Division’s  Automated 
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Management  System  (AMS)  and  the  Test  Wing's  Wing  Information 
System  (WIS)  could  distribute  information  faster  than 
current  methods.  Another  system  that  the  Test  Wing  manager 
said  would  have  been  useful  is  the  Computer  Supported 
Network  Analysis  System  (CSNAS).  It  would  have  been  used  as 
a  project  network  planner  of  major  program  tasks  and 
milestones.  The  problem  with  these  computer  management 
tools  is  the  necessary  time  required  and  difficulty  in 
maintaining  the  data  base  for  the  program.  The  benefit 
would  be  a  capability  for  managers  to  dial  up  program  status 
on  the  computer  system  as  required. 

The  SPO/Lab  manager  saw  some  things  differently  from 
his  'T'est  Wing  counterpart.  For  example,  he  said  he  would 
not  have  needed  a  computer  link  up  with  the  Test  Wing.  He 
would  have  preferred  more  informal  interchange  of 
information.  The  SPO/Lab  manager  said  that  the  Test  Wing 
manager  communicated  more  with  the  SPO/Lab  engineering 
office  than  with  the  SPO  test  office  or  the  program  manager 
himself.  He  felt  it  was  appropriate  for  them  on  this 
program  to  do  so  and  it  worked  out  well  since  the  final 
product  was  an  engineering  data  package. 

The  SPO/Lab  manager  commented  about  the  Test  Wing's 
negotiablity  by  saving  the  Test  Wing  was  entrenched  in  its 
way  of  doing  things.  He  felt  the  Test  Wing  could  have 
reported  more  of  what  he  expected  to  see  in  each,  category. 

A  specific  comment  about  cost  reporting  was  that  the  .JOCAS 
report  format  was  difficult  to  interpret.  He  also  said  the 
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program  overran  costs  by  a  significant  amount  due  to  the 
Test  Wing  taking  over  a  year  to  produce  the  final  report 
which  the  SPO/Lab  was  to  furnish  a  contractor. 

Answers  to  Research  Questions 

In  order  to  meet  the  research  objectives  stated  in 
Chapter  I,  specific  research  questions  were  developed.  The 
process  of  answering  the  research  questions  began  by 
formulating  the  methodoJogy  discussed  in  Chapter  III  of 
collecting  and  analyzing  perception  data  from  Test  Wine 
managers  and  their  SPO/Lab  counterpart  managers.  In  this 
section,  each  research  question  from  Chapter  I  will  be 
restated,  followed  by  a  general  response  from  the  analysis. 

Research  Question  No.  1 

What  are  the  current  categories  of  information  that  tlie 
4950th  Test  Wing  r epor t s - - f rom  the  S PO  management 
perspective  and  from  the  Test  Wing  management 
perspect ive? 

General  Response.  A  list  of  categories  of  information 
that  Test  Wing  managers  use  in  reporting  to  their  customers 
is  found  in  the  Test  Director's  Guide,  4Q50  "^ESTWP  80-1, 
Appendix  H  (80-1  :H-2).  Those  categories,  with  the  exceptiijn 
of  reporting  (which  was  excluded  from  the  interview  list) 
and  documentation/approvals  (referred  to  as  doc umen t a t i on  in 
the  interview  list)  are  as  follows:  pr oc ur emen t / s upp 1 y , 

legist,  os,  fiinding,  manpower,  documentation,  schedule, 
planning,  engineering,  safety,  mo d  i  f  i  c,  a  t  i  n  design,  cost, 
modification  i  n  s  t  a  1  1  a  t  i  on  ,  aircraft  ma  i  n  t  e  n.i  nc  e  ,  test  ifem. 


instrumentation,  flight  testing,  technical  performance 


range  support,  data  analysis,  demodification,  contractor 
interface,  and  prefab. 

From  this  list  of  22  categories  provided  to  the 
interview  respondents,  all  were  perceived  as  being  usad  by 
the  Test  Wing  in  reporting  information  to  their  .SPO/I.ab 
customers.  Those  categories  that  were  added  will  be 
addressed  in  the  general  response  to  the  next  research 
que  s  t i on . 

Research  Question  No.  2 

What,  if  any,  additional  categories  of  information  do 
the  Test  Wing  or  its  customers  believe  should  be 
reported? 

General  Response.  The  following  categories  were  added  bv 
the  interview  respondents  as  being  applicable  categories  to 
specific  progLdms;  however,  none  were  coincidentiy  added  bv 
counterpart  managers;  sub-contractor  interface,  fin.nl 
reporting,  flight  profiles,  ...base  plane,  tech  photc, 
demod  i  f  i  c  a  t  i  on  .equired  by  external  factors,  aiid  contrattor 
flight  training. 

Research  Question  No.  3 

What,  if  any,  categories  that  are  current  !■/  rep’C'rteJ  do 
either  the  Test  Wing  or  its  customers  f»;“el  shi'^uli.i  not 
be  reported? 

General  Re  s  p o  n  s  e  .  During  the  i  n  t  e  r  v  i  o  w  s  .  man  a  g  -a  r  s  w r 
asked  to  draw  a  line  rhcoon-h  arr/  of  the  oi'ogir  -o  'ho 

prose  r  i  Viod  1  i  '  t  r  ■  rn  *  (lo  J]  i  r'-.-  a  .f  _  Guppald'  h  '  h".,- 

felt  were  p.o't  appli'ihl"  to  rht;--ir  pro.  gram.  T'heio  woi-o  r.;-,' 


three  programs--C,  E  and  H--for  which  a  category  (prefab; 
was  identified  by  both  counterpart  managers  as  being  non- 
applicable.  The  only  other  category  counterpart  managers  of 
a  program  v Program  H)  did  not  identify  as  applicable  is 
demod i f i ca  t i on . 

While  the  category  of  prefab  was  not  perceived  as  being 
applicable  to  three  of  the  nine  programs,  the  remaining  two- 
thirds  of  the  managers  perceived  this  category  as 
applicable.  For  the  categorv  of  demodification,  counterpart 
managers  for  only  one  of  the  nine  programs  considered  it 
non -app 1 i ca t 1 e .  Therefore,  there  is  no  significant  reason 
why  the  categories  of  prefab  and  demodification  should  be 
deleted  ft om  the  prescribed  list. 

Research  Question  No.  4 

What  are  the  relative  weights  assigned  to  tiie  actual 

and  desired  categories  by  SPO/Lab  management  and  Test 

Wing  management? 

General  Response.  Each  interview  respondent  was  asked  to 
assign  a  level  of  "importance"  from  a  Likert  scale  'wiiere 
one  eqii ailed  little  importance  and  five  equalled  strone 
i  mp)0  r  t  a  nc  e  )  .  Mean  values  were  ('omputed  from  the  lesronses 
for  e  a  c  h  ;  a  t  e  g  o  r  v  and  are  presented  in  Table  2  o  f  t  h  i  c 
c  .h  a  p  t  e  r  .  a  condensed  v  e  r  s  i  o  n  of  T  i  b  !  e  2  is  p  r  e  s  e  n  ^  e  d  b  1  ■ '  w 
in  Table  LI.  T 1;  e  r  a  t  e  g ' '  r  i  °  s  in  Table  11  are  p  r  t  ‘  s  e  n  t  e  h 
a  c '■  Q  r  d  i  r;  y:  f  ■'  to'-  i  s  o  tid  ;  n  s  0.1  i  r  i  t'*  !  i‘  i'.'-- 

w  e  i  g  h  t,  s  . 


TABLE  11 


INFORMATION  IMPORTANCE  MEAN  RATING  VALUES 


CATEGORY  TEST  WING  CATEGORY  SPO/LAB 


FLIGHT  TESTING  5 . 0 
SCHEDULE  4.9 
TECH  PERFORMANCE  4.8 
PLANNING  4.7 
FUNDING  4.4 
COST  4.2 
TEST  ITEM  4  .  1 
ENGINEERING  3.9 
INSTRUMENTATION  3.9 
DATA  ANALYSIS  3.8 
CONTRACTOR  INT  3.8 
MOD  INSTALLATION  3.7 
SAFETY  3.6 
RANGE  SUPPORT  3  .  3 
DOCUMENTATION  3 .  1 
MOD  DESIGN  3  .  1 
DEMODIFICATION  3.0 
LOGISTICS  2.9 
A/C  MAINTENANCE  2.0 
PREFAB  2.8 
MANPOWER  2.7 


PROCUREMENT/SUPPLY  1.9 


SCHEDULE  4.9 
FLIGHT  TESTING  4  .  8 
FUNDING  4  .  3 
RANGE  SUPPORT  4  .  3 
PLANNING  4.2 
SAFETY  4.2 
COST  4.0 
TECH  PERFORMANCE  4  .  0 
DATA  ANALYSIS  4  .  0 
INSTRUMENTATION  3.9 
MANPOWER  3.8 
DOCUMENTATION  3.8 
MOD  DESIGN  3.8 
MOD  INSTALLATION  3.8 
CONTRACTOR  INT  3.8 
ENGINEERING  3.8 
TEST  ITEM  3.6 
LOGISTICS  3.3 
A/C  MAINTENANCE  3  .  2 
PROCUREMENT/SUPPLY  3.1 
PREFAB  3.0 
DEMODIFICATION  2.6 


The  mean  relative  weights  presented  in  Table  11  provide 
a  rank  ordering  of  the  categories.  The  table  shows  that  the 
SPOs/Labs  rank  more  cate.gories  above  3.0  than  the  Test  Wing. 
From  the  table  one  can  conclude  that  the  cate.gories  of 
schedule,  flight  testing,  funding,  and  planning  are 
perceived  Ijv  Test  Wing  managers  and  SPO/Lab  managers  t  be 
among  their  top  five  most  important  categories  in  whirh 
information  is  reported. 

P_e  s  e a  r  c  h _  Cue i  ')n  Ni)  .  5 

W  ti  a  ^  I  r t  .most  rr-mmon  methods  Test  Wing  manat’,>=‘rs  u.''e 
in  repr,rting  test  information  for  e.i-ti  r r  'j  t  t  j 


SPO/Lab  managers,  as  perceived  by  the  Test  Wing  and  by 

its  customers? 

General  Response.  The  methods  that  were  identified  by  the 
interview  respondents  are  the  following:  telephone, 

written,  group  meetings,  dialog  (i.e.,  face-to-face 
conversations  between  two  people!,  fax,  and  computer  link. 
Specific  written  categories  are  presented  in  Table  9  of  this 
chapter . 

In  the  interviews,  the  respondents  were  asked  to 
identify  the  most  commonly  used  method  of  communication  in 
reporting  information  for  each  applicable  category.  Very 
often  there  were  two  or  more  additional  methods  identified 
because  one  method  may  have  been  a  follow-up  to  the  primary 
method  (e.g.,  a  letter  following  a  telephone  call  to 
document  the  information  exchange). 

In  trying  to  identify  similarities  and  differences  in 
perceptions  of  Test  Wing  managers  and  their  counterpart 
SPO/Ldb  managers,  comparison  was  made  between  the  managers 
as  to  what  communication  method  they  each  thought  was  most 
commonly  used  in  reporting.  For  one  set  of  managers  (i.e.. 
the  counterpart  managers  associated  with  Program  G),  there 
was  a  59%  concurrence  in  responses.  In  other  words,  the  two 
counterpart  managers  identified  the  same  pri''^  ..ethods  of 
communication  for  13  of  22  categories.  Managers  from 
Program  G  had  the  highest  number  of  matched  responses  of  all 
the  programs. 
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Research  Question  No.  6 


What,  if  any,  differences  in  reporting  practices  would 

SPO/Lab  managers  recommend? 

General  Response.  The  following  is  a  compilation  of 
recommendations  from  the  open-ended  comments  of  the  SPO/Lab 
interview  respondents  about  what  the  Test  Wing  managers 
should  do  in  reporting  information: 

1.  Get  c 1  a r i f i ca t i on  from  SPO/Lab  manager  as  to  what 
information  he  or  she  expects. 

2.  Do  not  use  the  Job  Order  Cost  Accounting  System 
(JOCAS)  to  report  cost  information. 

3.  Report  cost  information  using  a  method  such  as  the 
telephone  rather  than  cost  reports  piesentlv  used. 

4.  Make  JOCAS  easier  to  interpret. 

5.  Submit  JOCAS  reports  that  have  current  cost 
estimates. 

6.  Provide  a  better  way  to  trad,  er^cpend  i  t  ures  ;  proviJ 
Cost  Schedule  Control  Svstem  Criteria  (C/SC SC)  reports. 

7.  Stay  informed  about  remaining  funds,  and  be 
prepared  to  report  status  to  the  customer. 

8.  De  more  flexible  in  meeting  customer  cost 
inform.ation  needs. 

9.  Be  more  timely  in  responding  to  customer 
information  needs. 

10.  Provide  more  detail  in  monthly  status  reports. 

11.  Play  an  active  role  in  reporting  to  the  SPO/Lab 


customer  even  though  a  contractor  may  be  conducting  the 
tests  and  reporting  information  to  the  SPO/Lab. 


12.  Remove  bureacratic  barriers. 

13.  Work  as  a  team. 

Most  of  the  recommendations  provided  by  SPO/Lab 
managers  were  regarding  cost  reporting.  The  comments 
expressed  the  managers'  discontent  about  the  Job  Order  Cost 
Accounting  System  (JOCAS)  and  their  desire  for  cost 
information  that  is  current  and  easy  to  interpret. 

Research  Question  No.  7 

What,  if  any,  differences  would  Test  Wing  managers 

prefer  to  make  in  reporting  information  to  their 

customers? 

1.  Compensate  for  inadequacies  in  formal  cost  reports 
(i.e.,  three  to  four  month  JOCAS  report  delays,  level  of 
difficulty  experienced  by  customers  in  i n t er pr e t a t i n g  actual 
costs  from  JOCAS  reports,  and  frequent  data  input  errors 
into  the  JOCAS  data  base)  by  providing  suplemental 
information  via  other  means. 

2.  Keep  customer  focal  point  informed  (i.e.., 
counterpart  manager)  when  communication  and  reporting  occurs 
with  various  members  of  the  customer's  matrix  or gan  i  za t  i  on  . 

3.  Assist  the  customer  with  Program  Introduction 
document  preparation  and  early  test  planning  activities  bv 
providing  a  Test  Concept  Document  (TCD).  The  TCD  is  a 
document  of  the  Test  Wing's  perspective  of  the  overall 
approach  to  the  test  program. 


Provide  information  to  SPO/Lab  customers  via  a 


^  . 

Local  Area  Network  (LAN)  using  the  existing  Automated 
Management  System  (AMS)  and  the  Wing  Information  System 
( WIS  )  . 

5.  Use  the  Computer  Supported  Network  Analysis  System 
(CSNAS)  as  a  project  network  planner  of  program  tasks. 

S  ummary 

This  chapter  presented  the  results  and  analysis  of 
perception  data  collected  from  Test  Wing  managers  and  their 
counterpart  managers  in  the  SPOs  and  Labs.  Comparisons  wer 
made  between  the  managers  responses  to  interview  questions 
in  the  following  four  areas:  information  importance, 

reporting  negotiability,  overall  reporting  pe;.  formance  .  and 
methods  of  communication.  Analysis  of  the  data  for  the 
first  three  areas  was  performed  using  descriptive  statistic 
■ind  statistical  tests  of  significance  to  identify 
similarities  and  differences  between  the  managers' 
responses.  The  fourth  area  was  analysed  by  listing  all  'lie 
methods  of  communication  by  program  and  by  o.ategorv  tn 
identify  similarities  and  differences  in  responses. 
Fr'iquency  counts  of  the  communication  methoils  wer'-‘  also  mi'l 
to  identify  s  i  m  1  1  a  r  t  i  e  s  and  differences  in  p  e  i‘  '  o  p  t  i '  a  i  s 
between  managers  about  which  methods  were  most  commonly  us" 
to  report  intormation.  Following  analysis  of  the  meftiod;-.  o 
commun  i  c.i  t  i  on  ,  there  was  discussion  on  the  issues  i*a  i  s^'i  !'■/ 
the  intyrvi''W  respondent::  in  their  op“n-end*‘d  ■  ''nunents. 

7B 


Problem  areas  and  a  few  recommendations  for  improved 
reporting  from  the  managers  were  recounted.  Finally,  the 
research  questions  were  answered  as  part  of  the  results  and 
analysis. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


V  . 


Over V i ew 

This  research  is  an  attempt  to  help  4950th  Test  Wing 
managers  determine  if  the  Test  Wing  is  responsive  in  meeting 
customer  information  needs.  The  following  research 
obiectives  were  formulated  to  logically  determine  the 
perceived  responsiveness  of  the  Test  Wing  in  meeting  the 
information  needs  of  its  SPO/Lab  customers: 

1.  to  determine  the  similarities  and  differences  that 
may  exist  between  the  perceptions  of  the  4950th  Test 
Wing  and  its  customers  concerning  what  constitutes 
effective  reporting,  and 

2.  to  determine  the  similarities  and  differences  tii.’t 
may  exist  between  the  Test  Wing's  assessment  ot  its 
performance  and  its  customers'  assessment  of  Test  Wing 
performance  in  reporting  information. 

Conclusions 

There  are  several  conclusions  that  can  be  made  as  a 
result  of  this  study  about  the  4950th  Test  Wing's  overall 
effectiveness  in  meeting  customer's  perceived  information 
needs.  The  first  conclusion  is  that  the  list  of  c,ateeoties 
of  information  already  used  bv  Test  Wing  managers  to  report 
Lest  program  progress  to  its  customers  is  .a  good  found. ation 
and  includes  most  of  what  constitutes  effective  test 
p  r  13  g  r  e  3  3  reporting.  This  list  is  f  <3  u  n  d  in  the  T  e  s  t 
Director's  C u i d e  .  4950  TESTP  80-1.  Appendix  H. 

The  second  conclusion  is  that  while  the  prescribed  list 
of  categories  i  .s  a  good  foundation  for  whtM  con.stitutes 
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effective  reporting,  it  is  not  totally  comprehensive. 
Managers  perceived  there  to  be  other  categories  in  which  to 
report  information  that  were  not  a  part  of  the  prescribed 
list.  It  is  good  that  the  Test  Wing  allows  flexibility  for 
its  test  managers  to  report  categories  outside  of  those  on 
the  prescribed  list.  Test  Wing  managers  should  be  =iware 
that  their  SPO/Lab  counterpart  managers  have  different 
perceptions  about  what  categories  of  information  are 
applicable  to  the  test  program,  and  they  sometimes  have 
different  expectations  about  what  should  be  reported. 

Reasons  why  there  are  differences  in  perceptions  about  what 
should  be  report  are  the  following: 

1.  Different  perceptions  about  what  the  categorv  of 
information  entailed  (although  care  was  taken  during  the 
interviews  to  convey  the  same  description  of  each  categorv 
as  shown  in  Appendix  B). 

2.  Varied  background.s  and  experiences  of  each  manager. 

3.  Varied  management  perspectives  due  to  the  nature  of 
each  manager's  perceived  test  management  role  (in  two  cases, 
the  Test  Wing's  specified  counterpart  S PO  manager  was  a 
program  manager,  not  a  test  manager). 

A.  Varied  perspectives  about  what  the  information 
requirements  of  the  test  program  were  or  would  be. 

5.  The  sensitivity  of  the  program. 

6.  Uniqueness  of  the  units  within  the  Test  Wing  that 
managed  the  program  or  reported  information  to  the  customer. 


7.  Varied  perspectives  about  what  information  the 
contractor  was  perceived  to  report  to  the  SPO/Lab  manager 
versus  what  the  Test  Wing  manager  was  expected  to  report 
(i.e.,  if  the  program  involved  contractor  managed  testing). 
Being  aware  of  these  reasons  for  differences  in  perceptions 
about  what  should  be  reported  could  be  valuable  to  Test  Wing 
managers  and  SPO/Lab  managers,  providing  them  with  insight 
about  their  counterpart  managers. 

The  Test  Wing  should  also  be  aware  that  it  is  sometimes 
perceived  as  not  being  very  flexible  in  negotiating  with  its 
customers  about  what  or  how  information  should  be  reported. 
Reasons  why  Test  Wing  managers  were  viewed  as  not  being 
flexible  are  the  following; 

1.  Personality  of  the  manager. 

2.  Method  of  communicating  (i.e.,  a  J0CA3  report  th.it 
was  often  described  as  useless;  however-,  it  is  still  used  irr 
reporting  cost  i nf orma t i on ) . 

3.  Entrenchment  in  doii'ig  things  a  certain  wav. 

<4  .  Political  factors. 

The  third  conclusion  from  t  li  i  s  sru'iv  was  ttr.rt 
differences  exist  in  how  much  Tost  Wing  managers  and  SPO/Lab 
managers  valire  the  i.mportance  c- f  the  r.itegories  of 
information  from  the  prescribe'!  list.  In  contrasting  the 
mean  Likert  scale  importance  ratings  between  counterp.rrt 
managers  for  each  category,  the  largest  d  i  f  f  e  lornc  e  s  occur 
for  the  categor  ies  of  r.ange  si4pp()rt.  ,  manpowf'r,  an'l 
pr  DC  u  r  emen  t  /  3  upp  1  y  .  The  Test  Wing  man.rgers  g..ive  e.ich  of 
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these  categories  a  lower  importance  rating,  which  means 
they  are  undervaluing  the  importance  of  these  categories  in 
relation  to  their  SPO/Lab  counterparts.  The  largest 
differences  for  which  the  Test  Wing  managers  overvalue 
categories  in  relation  to  their  SPO/Lab  counterpart  manager 
are  for  technical  performance  and  test  item.  Awarene.ss  ‘''f 
these  value  differences  may  alert  Test  Wing  managers  to  fin 
out  from  their  counterpart  managers  how  they  can  be  more 
responsive  in  reporting  information  in  these  categories,  it 
t  hey  apply . 

A  fourth  conclusion  of  this  study  is  that  some  T'‘-st 
Wing  customers  are  disgruntled  about  the  methci  in  which 
cost  information  is  reported,  and  the  Test  Wing  may  want  to 
consider  some  alternatives.  Several  comments  were  made  bv 
the  interview  respondents  about  the  Job  Order  Oost 
Accounting  System  (JOCAS)  describing  it  as  "poor", 
"worthless",  "difficult  to  understand",  "three  to  tout 
months  late",  "difficult  to  interpret",  and  "frequently 
[having]  data  input  errors."  One  course  of  action  for  tiu' 
Test  Wing  to  implement  in  resolving  the  perceived  cost 
reporting  problem  is  to  report  supplemental  ccst  informal io 
to  satisfv  the  customer's  cost  infornration  needs.  Tit  i  c 
alternative  has  been  used  effect ivelv  bv  some  Test  Wing 
managers.  .^noth-=‘r  course  of  .action  is  to  consider  using 
some  other  '^ost  reporting  system  that  mt'els  user  needs. 

While  there  are  courses  of  action  the  Test  Wine  mav 
want  to  follow  in  providing  better  servic,;  to  its  rvistomei  ;• 


there  are  also  areas  in  which  future  research  may  be 
profitable  for  managers  of  weapon  system  test  activities. 

Recommendations  for  Future  Research 

The  researcher  recommends  that  future  research  be 
conducted  in  the  following  test  management  areas: 

1.  Replicate  this  study  for  other  A950th  Test  Wing 
customers . 

2.  Replicate  this  study  for  other  Responsible  Test 
Organizations  (i.e.,  other  Test  Wings). 

3.  For  all  programs  where  4950tii  Test  Wing  customers 
use  mitrixed  personnel  resources,  interview  managers  and 
engineers  (other  than  the  Test  Wing  counterpart  manager)  who 
also  communicate  with  and  receive  reporting  from  the  Test 
Wing  manager.  These  interviews  would  seek  perceptions  about 
information  reported  bv  the  Test  Wing-  The  research  could 
also  identify  similarities  and  differences  in  whether 
information  received  by  the  primary  counterpart  SPO  manager 
from  the  Test  Wing  manager  is  identical  to  that  received  bv 
the  matrix  managers  from  the  Test  Wing. 

4.  Interview  ttie  ne:<t  level  of  managers  from  the  Tost 
Wing  and  the  SPOs/Labs  regarding  their  perceptions  about 
reporting  information  arid  compare  their  responses  with  their 
subordinates'  responses.  Research  could  be  conducted  to 
determine  if  the  supervisor's  management  views  are  imposed 
upon  subordinates  in  such  a  way  that  they  i  n  f  1  cienc 

3  u  b  o  r  d  i  n  a  t  e  -s  '  r  s  p  o  n  s  e  s  . 
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5.  Examine  the  effectiveness  and  suitabilitv  of  the 
Job  Order  Cost  Accounting  System  (JOCAS).  Develop  a  new 
cost  accounting/reporting  system. 

6.  Investigate  the  use  of  a  computerised  Local  Area 

Network  (LAN)  in  reporting  information  to  all  A950th  Test 

Wing  customers  at  Wr i gh t - Pa 1 1 er son  AFB,  Ohio.  Use  of  the 

Automated  Management  System  (AMS)  currently  in  place  at 

Aeronautical  Systems  Division  (ASD)  may  be  a  viable  vehicl 

for  reporting  information.  An  ASD  Reserve  Project  Report 

states  the  foliowing  about  AMS: 

.A  direct  support  from  AMS  will  eliminate  some  of  the 
redundant  reporting,  and  assist  in  assuring  the 
commonality  of  information  in  all  reporting  and 
analysis  syste.ms.  (3:12) 

7.  Investigate  the  use  of  a  test  program  planning 
system. 
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Appendix  A;  Scripts  of  Interview  Questions  Asked  of 
SPQ/Lab  Managers  and  Test  Wing  Managers 


Test  Wing  Version 


Introduct i on 

I'm  doing  some  research  for  the  4950th  Test  Wing's 
Directorate  of  Resource  Management,  Plans  and  Programs 
Division,  to  help  them  in  their  on-going  effort  to  main  tain 
and  improve  high  quality  reporting  to  Test  Wing  customers. 
Part  of  this  research  involves  collecting  information  from 
Test  Wing  managers,  like  yourself.  This  research  also 
includes  collecting  information  from  the  Test  Wing's 
customers  at  ASD  and  WRDC.  As  a  Test  Wing  manager,  vour 
expert  opinion  and  ideas  will  be  very  valuable  in  this 
research  effort.  Let  me  begin  by  asking  you  some  general 
questions  about  yourself  and  the  test  program  you  manage. 

1  I f  you  have  any  questions  during  any  portion  of  this 
interview,  please  feel  free  to  stop  me  and  ask). 


Demo.graphics 

1 . 

Date  of  interview: 

2  . 

Name  of  test  manager: 

3  . 

Rank/Grade : 

4  . 

Office  symbo  1  : 

5  . 

Name  of  test  program: 

6  . 

Length  of  time  as  a  test  manager: 

7  . 

Length  of  time  managing  this  program.: 

8.  Participated  in  original  PID/SC  negotiations 

9.  Current  phase  of  test  program  (i.e.,  testing, 
reporting,  completed): 

(  Y/N  )  ;  _ 

t  i  n  a  1 

10  . 
da  t 

Number  of  formal  test  progress  reports  submi 
e  ! 

t  t  e  d  r 

1  1  . 

Counterpart  test  manager  in  ASD  or  WRDC: 

12.  Number  of  times  per  week  you  are  in  contact 

SPO/Lab  counterpart: 

with  your 

Categories  of  Information 

Category  Identification:  This  research  involves 

identifying  categories  in  which  test  information  is 
reported.  The  ne.xt  few  questions  have  to  do  with  those 
categories.  I  want  to  ask  you  what  categories  of 
information  you  think  are  important  to  use  in  reporting  to 
your  5P0/Ldb  customer.  To  be  more  specific,  here  is  i  list 
of  categories  commonly  used  in  reporting  test  inform, it  ion 
(present  the  prescribed  list).  The  list  is  not  all 


inclusive,  and  you  will  notice  as  we  progress  through  the 
interview  that  in  some  cases  the  categories  overlap.  If  you 
need  clarification  on  any  of  the  categories,  I  can  provide 
you  with  a  description.  Please  look  over  the  list  and  draw 
a  line  through  any  categories  you  feel  are  not  or  will  not 
be  applicable  to  your  test  program  at  any  time  during  your 
work  with  your  customer.  You'll  notice  there  are  blank 
lines  at  the  bottom  of  the  column  for  you  to  add  any 
categories  you  feel  should  be  on  the  list. 

Information  Importance:  Next  I  would  like  to  get  a 
rating  from  you  as  to  how  important  vou  feel  these 
categories  are.  Please  assign  a  rating  to  each  category 
that  was  not  initially  crossed  out  by  using  this  scale 
(present  Likert  scale  1).  Notice  that  there  are  five 
ratings.  A  rating  of  five  means  that  vou  consider  the  it':m 
to  be  very  important  in  effectively  reporting  test  progress 
on  your  program.  As  the  scale  decreases  to  one,  the  item 
means  less  to  you  in  effectively  reporting  test  progress  on 
your  program.  Rate  each  of  the  categories  independently 
using  this  scale,  and  place  the  number  you  choose  on  the 
line  to  the  left  of  each  item. 

Reporting  Negot i abi ’ i ty ;  For  some  of  these  categories, 
you  may  have  had  an  opportunity  to  negotiate  wh.it  and  how 
your  counterpart  manager  expects  information  to  be  reported. 
Please  draw  a  line  through  those  categories  in  which  you  .ind 
vour  counterpart  manager  have  not  negotiated  reporting 
requirements.  Now  I  would  like  to  get  vour  perception  o 
how  flexible  in  negotiations  you  feel  you  were  in  meeting 
your  customer's  reporting  requirements.  Using  this  scale 
(present  Likert  scale  2)  and  the  list  of  categories  of 
information,  I  want  vou  to  rate  how  negotiable  you  feel  that: 
you  (the  Test  Wingl  were  in  meeting  the  information  needs  of 
your  customer  for  each  category.  As  you  did  before,  assign 
a  number  to  each  category  which  reflects  how  negotiable  v;.'u 
feel  that  you  (the  Test  Wing)  were.  A  rating  of  five  means 
that  you  were  extremely  negotiable.  As  the  scale  d  ec  r  ea  s 
in  number,  the  rating  reflects  that  you  were  less 
negotiable  for  whatever  r!;ason.  A  rating  of  one  means  th.it 
vou  were  not  negotiable  at  all.  Rate  each  of  the  c  .a  t  e  g'' r  i  e  s 
using  this  scale  and  place  the  number  vou  choose  on  the  line 
to  the  right  of  each  category. 

Overall  Reporting  Performance:  For  this  next  area,  I 
want  to  identify  all  the  categories  in  which  vou  have 
reported  some  information.  Please  draw  a  line  through  any 
category  in  which  you  have  not  reported  information  to  your 
counterpart  manager.  In  the  same  manner  as  before,  I  want 
you  to  assign  a  rating  for  how  well  you  feel  that  vou  (the 
Test  Wing)  have  done  overall  in  reporting  test  information 
in  each  of  the  categories.  Using  this  scale  (present  Likert 
scale  3),  assign  a  rating  to  each  category  by  putting  a 
number  on  the  line  to  the  right  of  each  item.  A  rating  ot 
five  means  you  feel  that  you  have  done  outst.inding  in 
repiarting  test  information  to  your  customer  for  that 


category.  A  rating  of  one  means  that  vou  fetrl  yi^u  liavo  ione 
poorly  in  reporting  in  that  category. 

Common i ca t i on  Methods:  Now  we  are  at  the  final  portion 
of  the  interview.  I  want  to  identify  how  you  have 
communicated  information  to  your  customer.  Please  tell  me 
what  you  think  is  the  most  commonly  used  method  of 
communicating  test  information  (whether  it  be  by  memorandom. 
formal  report,  telephone,  telefax,  etc.)  for  the  fol’ owing 
categories  (those  in  which  some  reporting  has  occurred I 
would  also  like  to  know  if  there  are  other  commun  i  c  a  t  i  u': 
methods  vou  use  for  each  category  as  well. 

Lastly,  if  vou  have  any  comments  about  the  met  b, o'..-  f 
communication  or  comments  about  reporting  in  general,  1  e  a 
let  me  know . 

This  now  concludes  my  formal  interview  questioning.  Do 
you  have  any  comments  or  questions  I  can  address? 


Prescribed  List  of  Categories  of  Information  ■  2  ;  H  -  2 


WEIGHT  CATEGORY 

’ROCUREMENT/.CUPPLY 

logistics 

r UNDING 

MANPOWER 

P'''CUME'nTATION 

•SCHEDULE 

PLANNING 

ENGINEERING 

SAFETY 

MODIFICATION  DESIGN 
COST 

MODIFICATION  INSTALLATION 
AIRCRAFT  MAINTENANCE 
TEST  ITEM 
INSTRUMENTATION 
FLIGHT  TESTING 
TECHNICAL  PERFORMANCE 
RANGE  SUPPORT 
DATA  ANALYSIS 
DEMODIFICATION 
CONTRACTOR  INTERFACE 
PREFAB 
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Likert  Scales 


Likert  Scale  1;  Importance  of  Categories 

"How  important  do  you  consider  this  category  to  be 
(independent  from  the  others)  in  reporting  information  to 
VO  ur  customer?” 


LESS 

VERY 

IMPORTANT 

IMPORTANT 

1 

2 

3 

4 

5 

Likert  Scale  2:  Reporting  Negotiability 

"How  flexible  in  reporting  negotiations  do  you  feel  vo 
were  in  meeting  your  customer's  perceived  information  needs 
for  this  category?" 

NOT  NEGOTIABLE 
AT  ALL 

12  14 


Likert  Scale  3;  OveralL  Reporting  Per fcjrmance 

"How  well  do  you  feel  you’ve  done  O'/erall  in  reporting 
information  f(jr  this  category  to  vour  customer?" 


POOR 

OUT.STANDING 

1 

2 

3 

4 

5 

EXTREMELY 

NEGOTIABLE 


Version 


I  n  t  r  oduc  t  i  o:. 

I'm  doing  some  research  for  the  4950th  Test  Wing  to 
help  them  in  their  on-going  effort  to  maintain  and  improve 
high  quality  reporting  to  Test  Wing  customers.  Part  of  thi 
research  involves  collecting  information  from  Test  Wing 
customers,  like  yourself.  This  research  also  includes 
collecting  information  from  the  Test  Wing’s  own  members.  A 
a  Test  Wing  customer,  your  expert  opinion  and  ideas  will  he 
very  valuable  in  this  research  effort.  Let  me  begin  bv 
asking  you  some  general  questions  about  yourself  and  the 
test  program  you  manage.  (If  vou  have  any  questions  during 
any  portion  of  this  interview,  please  feel  free  to  stop  me 
and  ask). 


Demographics 

1.  Date  of  interview; _ _ 

2.  Name  of  Lest  manager;  _ 

3 .  Rank/Gr ade  ; _ 

4.  Office  symbol  (include  ASD  or  WRDC); _ _ _ 

5.  Name  of  test  program;  _ 

6.  Length  of  time  as  an  A.3D/WRDC  test  manager  :  _ 

7.  Length  of  time  managing  this  program; _ _ 

8.  Participated  in  original  PID/SC  negotiations  (Y/N'-:  _ 

9.  Current  phase  of  test  program  (i.e.,  testing,  final 

reporting,  completed):  _ 

10.  Number  of  formal  test  progress  reports  received  from 

Test  Wing  to  date:  _ 

11.  Counterpart  test  manager  in  4950th  Test  Wing:  _ 

12.  Number  of  times  per  week  you  "re  in  contact  with  vciur 

Test  Wing  counterpart;  _ _ _ _ 

Categories  of  Information 

CategOi/  Identification*  This  research  involves 
identifying  categories  in  which  test  information  is 
reported.  The  next  few  quest'ens  have  to  do  with  those 
categories.  I  want  to  ask  you  what  categories  of 
information  vou  think  are  important  for  the  Test  Wing  to  use 
in  reporting  to  you.  To  be  more  specific,  here  is  a  list  of 
categories  (present  the  prescribed  list)  tTiat  are  commonlv 
used  by  the  Test  Wing  in  reporting  to  you.  The  list  is  not 
all  inclusive,  and  you  will  notice  as  we  progress  through 
the  interview  that  in  some  cases  the  categories  overlap.  If 
vou  need  clarification  on  anv  of  the  categories.  I  can 
provide  you  with  a  description.  Please  look  over  the  list 
and  draw  a  line  through  any  categories  vou  feel  are  not  or 
will  not  be  applicable  to  your  test  program  at  any  time 
during  your  work  with  the  Te.st  Wing.  You'll  notice  there 
are  blank  lines  at  the  bott-am  of  the  column  for  vou  to  add 
anv  categories  you  feel  should  be  on  the  list. 

Information  Importance:  Next  I  would  like  to  get  a 
rating  from  vou  as  to  how  important  you  feel  these 
categories  are.  Please  assign  a  rating  to  each  category 
that  was  not  initially  crossed  out  by  using  this  scale 
(present  Likert  scale  1).  Notice  that  there  are  five 
ratings.  A  rating  of  five  means  that  you  consider  the  item 
to  be  very  important  in  effectively  reporting  test  progress 
on  your  program.  As  the  scale  decreases  to  one,  the  item 
means  less  to  you  in  effectively  reporting  test  progress  on 
vour  program.  Rate  each  of  the  categories  independently 
using  this  scale,  and  place  the  number  you  choose  on  the 
line  to  the  left  of  each  item. 

Reporting  Negotiability;  For  some  of  these  categories, 
you  may  have  had  an  opportunity  to  negotiate  with  vour 
counterpart  manager  in  the  Test  Wing  as  to  wh.st  and  how  vow 
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expect  i  nf  o  r  ma  t ’■  on  should  be  reported  to  you.  Please  draw  a 
line  through  those  categories  in  which  you  and  your 
counterpart  manager  have  not  negotiated  reporting 
requirements.  Now  I  would  like  to  get  your  perception  of 
how  flexible  in  negotiations  you  feel  your  counterpart  Test 
Wing  manager  was  in  meeting  your  reporting  r e qu i remen t c . 
Using  this  scale  (present  Likert  scale  2)  and  the  list  of 
categories  of  information,  I  want  you  to  rate  how 
negotiable  you  feel  the  Test  Wing  was  in  meeting  your 
information  needs  for  each  category.  As  you  did  before, 
assign  a  number  to  each  category  which  reflects  how 
negotiable  you  feel  the  Test  Wing  was.  A  rating  of  five 
means  that  they  were  extremely  negotiable.  As  the  scale 
decreases  in  number,  the  rating  reflects  that  your 
counterpart  manager  was  less  negotiable  for  whatever  reason. 
A  rating  of  one  means  that  your  counterpart  was  not 
negotiable  at  all.  Rate  each  of  the  categories  using  this 
scale  and  place  the  number  you  choose  on  the  line  to  the 
right  of  each  category. 

Overall  Reporting  Performance;  'or  this  ucxl  area.  I 
v/ant  to  identify  all  the  categories  in  which  you  have 
received  some  information  from  your  counterpart  manager  it 
the  Test  Wing.  P  Lease  draw  a  line  through  .any  categ<3r'.'  in 
which  you  have  not  received  information  from  voui 
counterpart  manager.  In  the  same  manner  as  before,  I  w-int 
you  to  assign  a  rating  for  how  well  vou  feel  the  Test  Wing 
has  done  overall  in  reporting  test  information  in  e.^ch  f 
the  categories.  Using  this  scale  (present  L’ke:'^  sc.\1l'  ^  )  . 

assign  a  rating  to  each  category  by  putting  a  number  on  the 
line  to  the  right  of  each  item.  A  rating  of  five  means  V'u 
feel  your  counterpart  manager  has  done  outstanding  in 
reporting  test  information  for  that  category.  A  rating  f 
one  means  that  vou  feel  yoi.jr  counterpart  has  done  poorlv  in 
reporting  in  that  category. 

Communication  Methods;  Now  we  are  at  the  final  portion 
of  the  interview.  I  want  to  identify  how  your  counterpart 
manager  in  the  Test  Wing  has  communicated  information  to 
you.  Please  tell  me  what  vou  think  is  the  most  commonly 
used  method  of  communicating  test  information  (whether  it  be 
by  memorandom,  formal  report,  telephone,  telefax,  etc.')  for 
the  following  categories  (those  in  which  some  reporting,  has 
occurred).  I  would  also  like  to  know  if  there  are  other 
communication  methods  vou  use  for  each  category  .as  well. 

Lastly,  if  you  have  any  comments  about  the  methods  of 
communication  or  comments  about  reporting  in  general,  please 
let  me  know . 


Prescribed  List  of  Categories  of  Information 


Same  as  Test  Wing  version. 


Likert  Scales 


Likert  Scale  1:  Importance  of  Categories 

"How  import  a nf  do  you  consider  this  category  to  be 
(independent  from  the  others)  in  reporting  information?" 


LESS 

V  SPY 

IMPORTANT 

IMPORTANT 

1 

3 

A 

Likert  Scale  2:  Reporting  Negotiability 


"How  flexible  in  reporting  negotiat 
Test  Wing  was  in  meeting  your  perceived 
for  this  category?" 

ions  do  vou  feel  the 
information  nee 'is 

NOT  NEGOTIABLE 

AT  ALL 

EXTREMELY 

NEGOTIABLE 

1 

2  3 

A  5 

Likert  Sc^le 

Overall  Reporting  Performance 

"How  well  do  vou  feel  the  Test  Wing  has  done  overall  in 
reporting  information  to  vi?u  f'.-'t-  this  category?" 

POOR  nUTSTAMDIMC 

12  3  A  5 


Appendix  B:  Description  of  Terms  for 

Categories  of  Information 

Descriptions  for  the  categories  below  were  deriveii  from 

the  researcher's  understanding  of  the  terms  and  a 

conversation  with  the  point  of  contact  for  Appendix  H  of  f  h. - 

Test  Director ' s  Guide ,  4950  TESTWP  80-1  m_5;2:H-2). 

Procurement  /S  up  p  1  V'  ■  Purchases  or  loan  of  equipment  and 
parts  which  may  impact  the  test  program  progress  (e.g.,  lo>nr 
lead  i terns ) . 

Logistics.  Provision  and  maintenance  of  resources  in 
support  of  flight  testing.  This  category  includes  the 
transportation  of  equipment  and  movement  of  personnel 
supporLing  the  test  effort.  The  only  area  of  logistics 
excluded  from  this  broad  category  in  this  research  is 
a  i  r  c  r  a  f  t  m  a  i  n  t  e  n  a  n  c  e  . 

F  und  i  ng  .  Allocation  of  funds  to  the  Responsible  Test- 
Or  gan  i  za  t  i  on  by  the  .'ivstem  Program  Office  or  La  ho  r  .i  :■  ■■  r  v  . 
Information  reported  could  be  notification  of  a  de  f  i  c  i  en'".' 
of  funds  or  overage  of  funds. 

Manpower  .  Availability  of  people  to  accomplish  the  pregrain 
and/or  test  mission  (e.g.,  other  priorities  m  a '/  take 
resources  and  thus  delay  the  program). 

Do  c  umen t a  t i on .  Information  involving  more  than  just  written 
reports.  For  e.'ample,  information  regarding  engineering 
drawings  and  processing  of  airworthiness  certificates  and 
environmental  certificates  are  reported  in  this  category. 

S  c  h  e  il  u  1  e  .  Information  about  general  impacts,  advances, 
and/or  delays  in  the  established  mi]est'''nes  and  inchstoneo- 
of  the  program . 

P 1 ann i ng .  The  strategy,  considerations,  and  courses  of 
action  in  successfully  accomplishing  the  program. 

Information  commonly  reported  includes  a  defect  in  or  lack 
of  planning. 

Engineering .  Integration  of  the  t.  ec  hnical  aspects  of  the 
specific  tests  to  be  accomplished  during  flight  testing. 

Safety.  Concerns  about  the  safe  conduct  of  the  te.sts  (  i  .  . 

safety  of  flight  concerns,  safety  certifications  for  the 
modifications  and  equipment,  mishap  avoidance,  etc). 
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Modification  Design.  Form  and  fit  design  con'crris 
create  problems  in  which  a  component  is  not  designed  . 
properly  interface  with  other  component ( 3 ) . 

Cost.  Estimates  and  actual  dollar  expenditures  incurred 
the  Responsible  Test  Organization. 

Modification  Installation  .  M  a  n  i  f  e  s  t  a  t  i  o  n  of  form  or  f  : 
problems  during  the  installation  of  t h^  modifica' i ■  n . 
Component  interface  problems  may  arise  involving  -h': 
aircraft,  the  modification,  special  mountings,  eguipmen* 
p  a  1 1 e  t i z  a  t i o  n  ,  etc. 

Aircraft  Maintenance.  Planned  or  unplanned  occ  ur  a  nc  s 
downtime  for  the  aircraft  to  be  inspected/serviced  0; 
r epa i r ed . 

Test  I  tern .  The  physical  device! s)  to  be  tested. 

I  ns  t  r  umen  t  a  t i on .  Equipment  installed  in  the  aircraft  uc 
to  measure  and  record  test  data  involving  the  test  i  >•  eno 

Flight  Testing.  Actual  conduct  of  tests  and  oillecti.r. 
appropriated  data  during  a  flving  mission. 

Technical  Performance.  Operation  or  functioning  t  t 
Wing  with  regard  to  the  -effectiveness  f  the  t-'st.'. 

Range  S  lippo  r  t  .  Support  of  resources  that  proviife  tlie 
physical  space  to  test,  targets,  simulated  andZ-'r 
threats,  •3upp(?it  aircraft,  etc. 

Da  t  a  Ana  1  vs i s .  Manual  or  computerized  determination  c f 
results  (i.e.,  the  t  r  ans  f  or  ma  t  i.  on  of  flight  test  liat.!  i  .'i 
useful  1 n  f  o  r  m  a  t i o  n  i  . 

Pernod  i  f  i  c  a  t  i  on  .  Removal  of  the  test  item  and  its  ac.'-'o-i 

e qu i pmen t  . 

Contractor  Interface.  Interaction  between  Governrn-.'n ' 
personnel  and  contra-rt-jr  employees  involved  witn  th--^ 
test i n  g . 

Prefab .  Prefabricated  test  item  components  or  test  item 
support  equipment  (i.e.,  rad-'-mes  ,  cables,  racks,  wingtip 
etc). 
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which  they  can  report  test  program  information  to  their 
customers.  Customers  such  as  the  Aeronautical  Systems 
Division  System  Program  Offices  and  the  Wright  Research  and 
Development  Center  Laboratories  differ  from  Test  Wing 
managers  in  their  perceptions  about  which  categories  are 
applicable  to  their  programs.  Test  Wing  managers  and  their 
customers  value  the  importance  of  the  categories  of 
information  similarly  for  most  of  the  categories.  Although 
many  perceptions  about  the  category  of  information 
applicability  and  importance  may  be  similar,  differences  do 
exist  and  for  a  variety  of  reasons. 

The  Test  Wing  was  rated  by  itself  and  its  customers  in 
how  flexible  they  were  perceived  to  be  in  negotiating  what 
and  how  information  for  each  category  was  to  be  reported  to 
customers.  Results  showed  that  Test  Wing  managers  and  their 
customers  have  similar  perceptions. 

In  rating  Test  Wing  overall  reporting  performance  for 
each  category  of  information,  Test  Wing  managers  and  their 
customers  have  similar  perceptions. 

Open-ended  comments  by  the  interview  respondents 
suggested  reform  in  the  methods  in  which  the  Test  Wing 
reports  cost  information. 
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